Forgotten Backup

A file was excluded from search indexing but left on the web server. Determine whether discouraging discovery also prevents direct access.

Objective

• Use a standard discovery file to locate an unlinked resource.
• Distinguish crawler guidance from authorization.

Before you begin

Open /challenge/robots and keep all requests on the Pocket CTF host.

Concepts

robots.txt asks cooperative crawlers not to visit selected paths. It is a public hint, not an authentication or authorization mechanism.

Suggested tools

• Browser or HTTP client: Either can retrieve short text resources.

Tool practice

Rehearse the worked tool exercise before using challenge data.

Enter a root-relative path after the host, not after the current challenge path. In the returned text, a colon separates a directive from its value; retain the leading slash when following a disclosed path. Record both the discovery response and the resource response.

Investigation

Observe

Retrieve the conventional crawler-policy file from the site root and record each excluded path.

Form a hypothesis

Does the server reject direct requests to an excluded file, or merely omit it from normal navigation?

Test carefully

Follow only the path explicitly disclosed by the training site.

Progressive hints

Troubleshooting and reset

Use an absolute path beginning with /. Both resources are plain text and do not require a session reset.