This article focuses on a topic that is near and dear to my heart: logs. Whether the logs are application, infrastructure or device logs they are all important; critical even, I would argue and I’d hope you’d agree. Have you ever considered logs to potentially accumulate debt much the same as any other system.
Recently there was an email sent out by Github relating to a number of users being affected by a bug within the NPM system in that it logged passwords in plaintext in the logs. They do state that the exposure was limited in that it was only exposed to users within Github that had access to the logs. The blame was placed on a bug that stored the passwords in internal logs. The passwords which were logged were those from users that initiated a password reset. The company stated that they found this during a routine audit. If you are in the business of manufacturing devices, or software you should take some time to consider your logs. Indeed, I know this is not the most thrilling prospect. However, logs are certainly going to be your saving grace during an incident or your downfall in it causing an incident. This means that logs should be considered as part of the design process. There are certainly steps you can take to better design, implement and maintain your logs.
Logs are not only for the purpose of security or debugging. There are, in fact, a plethora of uses for logs outside of security and debugging:
- Business analytics: Using logs to gather valuable business related information to make informed business decisions.
- System and Performance analytics: Using logs to determine whether the system is performing as expected, and see whether there are deviations from the norm.
- In-field investigations: This may be a form of debugging, however when customer complaints are received from the field, logs are the first point of call to determine why something is not behaving as it should.
- Audits: This is when logs are used during auditing processes to determine whether the system is behaving as it should.
- Development Logs: These are logs that are used during the process of development and are more to fulfil the process of debugging, and ensuring that the code is producing the expected outcome. These are similar to logs within the console.
- Logs for Early Detection: Finding a potential incident early on allows you to effectively respond much quicker and more effectively. It allows for you to become proactive versus reactive.
The review of your logs after every major update could potentially save you from the embarrassment of having data found by an external party, especially when it is a component to a potential compromise. There are things that increase the probability of an accidental disclosure. To understand what sensitive data is one can define it as information that contains the following:
- passwords
- usernames
- session IDs
- source code
- encryption keys
- endpoints
- JWT tokens
- Personal health information (PHI)
- Personally Identifiable Information (PII)
This is not an exhaustive list but a start. As an organisation you should clearly define what is considered to be sensitive. The consideration should be given to not give away information to attackers or potential attackers if they gain access to logs. As a hacker this is the first treasure trove that I would delve into, it seems that logs are important to all, and so much more than just for debugging and security. There are things that should be avoided to ensure that your log health remains good and your logging debt remains low.
- Create an effective logging strategy that can be pushed through the organisation. The first step to observability is to create logs which can be enriched and used for analytical processes, for the team, business, security and incident response.
- Consider where your logs will be ingested into. This largely determines which types of logs you should design and how you should design them. It is generally better to go native versus not. For example if you can use syslog it is a good option, if you are developing for iOS then use the native logging framework for iOS, which is very good.
- You should do impromptu log analysis of your logs. While the collection and the storing of the logs is critical it is worthless if you do not do anything with them. Doing that first part is half a step in the right direction. These logs should be monitored and reviewed. This allows an organisation to shift from being reactive to proactively finding events, problems or sensitive information in their logs. This changes the dynamic to a more proactive one.
- You should determine how long your retention period should be for the logs. The biggest frustration for any security team, and for incident responders is function under the premise that they have the logs to investigate an incident only to have the realisation that the logs are no longer available due to a retention policy. Taking into account that there is a long duration between when an incident has taken place and when it has been detected might be more than 60 days. In fact 60 days might be fairly optimistic. Your retention of log data should exceed that of your detection rate.
- Logs should be standardised across the organisation. This means that logs should be universally the same across an organisation. Lack of standardised log formats can lead to logs being difficult to parse. To avoid death by parsing, ensure that regardless of the source of the log that it is normalised across the organisation. This makes the collecting, ingestion, and enrichment process an easier one. It also allows for logs not to be specialised in that only one member of the team can understand them. The time and effort that goes into building custom parsers could be better spent on actual features or improving your logs. Your logs should be simple and clear enough that the workflow can be understood by anyone.
- Avoid the dreadful “printf“. Sensitive data is often introduced through the use of “printf“. It is understood that this is often used during the process of debugging, and processes in which the response is needed. However one should be really careful with this as it can cause you headaches down the line.
- Avoid full response bodies and headers. Response bodies are another one of those sneaky things used by developers for debugging purposes that can be helpful to you, but when you look away it can trip you up and cause someone to open a window that you thought was closed, and bolted down. Do you not give red teamers free information, make them work for it. These bodies of text often contain secrets or inside information that is best not included.
- Knowing the normal to know the abnormal; this is a saying that I live by, it is not mine however, I was taught this on a SANS course. It rings true for everything in life. As a digital forensic practitioner we are pattern oriented. That being said, log analysis is simply knowing what the normal workflow looks like, and looking for deviations. Logs are great for a great many things but what they are truly amazing for is that they give us a look under the bonnet of our device, application or infrastructure at how they are performing. This allows organisations to look for real anomalies in log data, beyond “known bad” and “not known good.”
This list is not an exhaustive one. Logs, I think we can agree on, are an important component to all. It is often the ugly stepchild that most developers and security folks only look at when it makes a noise or something has been broken. Logs are something you should consider in the beginning of the design process. Nothing worse exists than an incident responder stating if only we had this logged, build in the information with the future in mind. Logs are a rich source of information that allow you to look under the hood.
In my next article I will discuss some approaches that can be taken to design your logs better. I’ll say this again: if you neglect your logs inevitably you might find yourself on the other side of a disclosure of sensitive data. It is better that you find what is wrong with your logs, before someone else does. Check your logs regularly, and reward those that find the disclosures and fix them.