Logging Debt: Is Your Logging Costing You More Than You Think?
Logs will be your saving grace during an incident, or the cause of one.
This article is about a topic close to my heart: logs. Application, infrastructure or device logs, they all matter. Critical, even, I would argue, and I hope you agree. But have you ever considered that logs can accumulate debt, just like any other part of a system?
GitHub recently emailed a number of users affected by a bug in the npm system that wrote passwords to internal logs in plaintext. The passwords belonged to users who had started a password reset. GitHub said exposure was limited to employees with access to those logs, and that the issue was found during a routine audit.
If you build devices or software, take some time to consider your logs. It is not the most thrilling prospect, but logs belong in the design process, and there are concrete steps you can take to design, implement and maintain them better.
Logs are not only for security and debugging
There is a whole range of other uses for them:
Gathering business information to support informed decisions.
Checking that the system performs as expected and spotting deviations from the norm.
When customer complaints arrive from the field, logs are the first place to look for why something is misbehaving.
Evidence during audit processes that the system behaves as it should.
Used while building and debugging to confirm the code produces the expected outcome, much like console output.
Spotting a potential incident early lets you respond faster and more effectively, and moves you from reactive to proactive.
Know what sensitive data looks like
Reviewing your logs after every major update could save you the embarrassment of an external party finding your data, especially when it becomes part of a compromise. Sensitive data includes, at a minimum:
passwords
usernames
session IDs
source code
encryption keys
endpoints
JWTs
PHI
PII
This is a start, not an exhaustive list. Every organisation should define clearly what it considers sensitive, so that logs give nothing away to anyone who gains access to them. As a hacker, logs are the first treasure trove I would dig into.
Keeping your logging debt low
Practices that keep your logs healthy:
Push it through the whole organisation. The first step to observability is logs that can be enriched and used for analysis by the team, the business, security and incident response.
The destination largely determines which logs you need and how to design them. Going native is generally better: use syslog where you can, and on iOS use Apple’s native logging framework, which is very good.
Collecting and storing logs is only half a step. Monitor and review them, including impromptu analysis, so you find events, problems and sensitive data before someone else does.
Few things frustrate incident responders more than discovering the logs they need have been deleted by a retention policy. The gap between compromise and detection can easily exceed 60 days, and even that may be optimistic. Your retention should outlast your time to detect.
Inconsistent formats lead to death by parsing. Normalise logs regardless of source so collection, ingestion and enrichment stay simple, and so no single team member is the only one who understands them. Time spent on custom parsers is better spent on features or on better logs.
printfSensitive data often slips in through quick debug print statements. They are useful during development, but be very careful, because they cause headaches later.
Another debugging habit that helps you today and trips you up tomorrow, reopening a window you thought was closed and bolted. They often contain secrets or internal detail. Don’t give red teamers free information; make them work for it.
A saying I live by, which I learned on a SANS course. Forensic practitioners are pattern-oriented, and log analysis is knowing what the normal workflow looks like and looking for deviations. Logs let you look under the bonnet of a device, application or infrastructure, and find real anomalies beyond “known bad” and “not known good”.
Design with the future in mind
This list is not exhaustive. Logs are often the neglected corner that developers and security teams only look at when something makes a noise or breaks. They belong at the start of the design process. There is nothing worse than an incident responder saying “if only we had logged this”.
In my next article I will look at approaches for designing your logs better. Until then, I will say it again: neglect your logs and you may find yourself on the wrong side of a sensitive data disclosure.
