Information Disclosure – Sensitive Resource Exposure via robots.txt | Sundial Observatory
Lab Link
Lab: Sundial Observatory
Overview
Sundial Observatory maintains a public website containing information about astronomy events, member activities, and club operations.
The site’s administrator attempted to hide sensitive content by listing internal paths inside a robots.txt file. Unfortunately, robots.txt is publicly accessible by design and merely provides guidance to search engines.
Because the protected content is still directly accessible, attackers can use the information disclosed by robots.txt to discover sensitive resources.
This results in information disclosure through exposed administrative and member-only paths.
Objective
Leverage information disclosed through robots.txt to locate a sensitive page and retrieve the flag.
Vulnerability Identification
This challenge is primarily an Information Disclosure vulnerability.
Classification Hierarchy
A02 - Security Misconfiguration └── Sensitive Resource Exposure └── Information Disclosure └── Sensitive Paths Revealed via robots.txt
Reconnaissance
The first step during web application reconnaissance is often reviewing publicly accessible files.
One commonly overlooked file is:
1
/robots.txt
Navigate to:
1
https://10536283-4065-occultation-469bd.challenges.webverselabs-pro.com/robots.txt
The application returns:
1
2
3
Disallow: /members-only-2026
Disallow: /staff-archive
Disallow: /old-roster.html
These entries immediately reveal potentially sensitive locations.
Exploitation
Step 1 - Review robots.txt
Access:
1
/robots.txt
Contents:
1
2
3
Disallow: /members-only-2026
Disallow: /staff-archive
Disallow: /old-roster.html
Many administrators incorrectly assume that listing paths in robots.txt hides them from users.
In reality, the file publicly advertises those locations.
Step 2 - Test Disclosed Endpoints
Visit the first disclosed endpoint:
1
https://10536283-4065-occultation-469bd.challenges.webverselabs-pro.com/members-only-2026
A secure application would return:
1
403 Forbidden
or
1
2
302 Redirect
Location: /login
Instead, the page loads successfully.
Step 3 - Confirm Information Disclosure
The supposedly private member area is fully accessible without authentication.
This confirms that sensitive content has been exposed through a publicly available resource.
The issue is not that robots.txt exists.
The issue is that sensitive pages remain accessible while relying on robots.txt as a security mechanism.
Step 4 - Retrieve the Flag
Scroll to the bottom of the page.
The flag is disclosed directly within the member content.
1
WEBVERSE{.....}
Proof of Exploitation
robots.txt
1
2
3
Disallow: /members-only-2026
Disallow: /staff-archive
Disallow: /old-roster.html
Sensitive Page
1
/members-only-2026
Result
1
Sensitive content accessible without authentication
Flag
1
WEBVERSE{.....}
Impact
An attacker can:
- Discover hidden resources.
- Access internal pages.
- Enumerate administrative functionality.
- Locate archived content.
- Expose sensitive information.
- Identify additional attack surfaces.
In real-world environments, robots.txt files have exposed:
- Admin panels
- Backup directories
- Development environments
- Internal documentation
- Staging applications
- Sensitive reports
Mitigation
Do Not Use robots.txt for Security
robots.txt is an indexing directive, not an access control mechanism.
Implement Proper Authorization
Sensitive resources should require:
1
2
Authentication
Authorization
before access is granted.
Remove Sensitive URLs from Public Files
Avoid listing confidential resources in:
1
2
3
4
robots.txt
sitemaps
JavaScript
comments
Restrict Access Server-Side
Protected pages should return:
1
403 Forbidden
or redirect users to authentication workflows.
Perform Reconnaissance Reviews
Regularly inspect publicly exposed resources for unintended disclosures.
Real-World Insight
Security assessments frequently begin with reviewing:
1
2
3
4
5
robots.txt
sitemap.xml
.git
backup files
source code comments
because organizations often leak sensitive information through these locations.
The misconception that robots.txt provides security has existed for decades. Search engines may respect its instructions, but attackers read it specifically to discover content that administrators do not want indexed.
The Sundial Observatory challenge demonstrates an important lesson:
If a page must remain private, access control must protect it. Hiding a URL is not security.