Scroll down to see comparisons of two users working on the same task
Joe gets a call from support
Some users are complaining that they cannot upload files
Mike gets a call from support
Some users are complaining that they cannot upload files
Joe checks the logs
Joe knows that his team did not deploy anything recently. He starts looking at the logs to see what is
going on.
Mike checks fluxroll
Mike knows that his team did not deploy anything recently. He checks fluxroll to see what changed.
Mike finds a possible cause
Because fluxroll tracks dependencies, Mike can see that a new configuration
was recently deployed for the HTTP servers running UploadService a service that his application
depends upon.
Mike confirms the root cause
Mike looks at the change using fluxroll. He sees that the configuration change restricted
file uploads to 2MB as compared to the earlier limit of 5MB. This explains why only some users were
affected.
Joe finds a problem in the logs
The logs indicate that UploadService, a service his application depends upon, is failing for some users.
Joe wants to understand why this only affects some users.
He starts searching through his company's repositories to see if he can find the code for UploadService
and see if it recently changed.
Mike locates the responsible person
Mike checks fluxroll to see who is on call for the HTTP server. He calls the person, explains the
situation and they decide to revert the configuration.
The change is rolled back
Everything is back to normal!
Joe finds the codebase
Joe finally finds the codebase for UploadService. However no changes were made to it recently. To help
him investigate this further, Joe asks the support team to contact Ben. Ben is the last person who made
a code change to UploadService. Joe is hoping that Ben can help him investigate the issue further.
Support team contacts Ben
The support team is able to reach Ben. Ben says he will help investigate the issue.
A conference bridge is created so Joe and Ben can work together with the support team.
Joe and Ben start investigating the issue
Joe and Ben try to reproduce the issue. They see no problem when they try uploading a file. They try
renaming the file, changing the extension, but still cannot reproduce the issue. The logs show nothing.
They finally ask support see if they can get a sample file from a customer who is experiencing the
problem.
The support team sends Joe and Ben a file
Joe and Ben notice that this file fails to upload. They take a few minutes but eventually determine that
the size of the file is the culprit.
Ben discovers the root cause
Ben notices that requests for the large file uploads don't even show up in his application logs.
This is only possible if the HTTP server directly rejects the request. He connects to one of the servers
running his application and notices that the configuration restricts file uploads to 2MB. He needs to
reach someone on the team that manages the HTTP servers.
Ben asks the support team to locate an engineer responsible for the HTTP servers
The support team takes 10 minutes and locates Marc. Marc is on his way to work but confirms that his team
had recently deployed a change to the HTTP servers that limited the size of file uploads.
He will revert the change when he gets in the office. ETA is 8am.
Marc reverts the changes Marc reverts the changes. Joe and Ben confirm that the configuration change fixed the problem.
The change is rolled back
Everything is back to normal!