Railway says about 100,000 workspaces may have had secrets exposed

Railway says an October security incident may have exposed customer environment variables after an unauthorized third party gained access to a subset of its internal systems.

Railway first disclosed the incident on Oct. 21, saying it had detected anomalous activity involving systems that store customer environment variables. Environment variables can include API keys, database credentials, private registry credentials, and other secrets used by deployed applications.

In its preliminary report, Railway said it became aware of a sustained, targeted campaign on Oct. 20. The company said the attacker appeared to be searching for AI provider and platform API keys, with early investigations indicating the AI provider keys were sought for distillation attacks.

Railway’s current timeline describes several key points:

  • Oct. 20, 13:58 UTC: Monitoring systems alerted Railway engineers to abnormal activity, initially seen as elevated latency.
  • Oct. 20, 14:03 to 16:51 UTC: Railway identified the activity as an attempted privilege escalation by a hostile external party and halted build pipelines to protect users.
  • Oct. 21, 04:24 to 05:23 UTC: Railway says the attacker was able to pull secrets in encrypted form, but was not able to decrypt them at that time.
  • Oct. 21, 05:27 to 09:18 UTC: Railway says the attacker accessed credentials tied to its CI environment and later made an unauthorized query of encrypted environment variables, with a subset decrypted using compromised credentials.
  • Oct. 22, 03:32 UTC: Railway says this was the point when it confirmed the attacker no longer had access to any internal systems.

Railway said it engaged external incident response experts and authorities to assist with forensics.

How many customers may be affected

Railway says at most 1.85% of all workspaces had some environment variables potentially compromised. It described that as about 100,000 workspaces.

The company is contacting customers it believes were directly impacted and customers it considers high risk. It also said the number of users who saw vendor-side key revocations may be larger than 100,000, because Railway proactively provided some vendors with broader key lists to reduce possible exposure.

Railway said a vendor notification about revoked keys does not automatically mean a workspace was impacted. Some providers were asked to revoke keys as a precaution, including GitHub, GitLab, Anthropic, OpenAI, OpenRouter, SpaceXSI, Stripe, and others.

Railway also said individual user accounts were not logged into as part of the incident. It has not ruled out source-code access and said it will address that point in its final forensic report.

The risk extends beyond Railway

If an exposed environment variable contained an AWS key, Google Cloud service account key, GitHub token, Stripe secret key, database password, or AI-provider API key, the risk follows that credential into the connected service. Railway’s post-incident guidance reflects that pattern, directing customers to check cloud logs, provider dashboards, billing records, repository access, and database sessions in addition to rotating values inside Railway.

That does not mean every stored credential was abused. The safest remediation path is to treat affected secrets as exposed until rotated and checked against provider logs.

Builds and deployments have largely returned

Railway paused builds and deployments during the incident response. Its preliminary report says existing user deployments stayed online throughout the incident, while customers experienced interrupted builds and deployments as well as periods of dashboard, API, CLI, and support unavailability.

Railway’s status page reported on Oct. 22 that builds and deployments for Enterprise, Pro, Hobby, Trial, and Free workspaces had returned to approximately normal operation. Railway also warned that commits pushed while deployments were paused were not queued, so affected services need a new deployment to receive those changes. Later updates on Oct. 23 said the investigation was still ongoing and Railway was still reaching out to customers it believes are at risk.

New signups remained temporarily paused in Railway’s Oct. 23 updates while the company focused on existing customers. Railway also said API tokens and Project Tokens created before the security incident remained invalidated, meaning customers using Railway’s Public API or CLI with those tokens need to create replacements.

What Railway customers are being told to do

Railway is recommending that all customers who signed up before Oct. 21 revoke and replace sensitive environment variables stored on the platform. The company has also published separate guides for cloud providers, databases, GitHub, Stripe, OpenAI, Anthropic, OpenRouter, Grok, and other services.

The main actions Railway is asking customers to consider include:

  • Revoke and replace sensitive environment variables and API keys stored in Railway before Oct. 21.
  • Rotate database credentials for Railway-managed databases and redeploy every service that consumes the rotated values.
  • Remove public database access when it is not required, especially while rotating credentials.
  • Review AWS CloudTrail, Google Cloud Audit Logs, GitHub logs, Stripe request logs, and AI-provider usage dashboards where relevant.
  • Redeploy services that use Railway storage bucket credentials, because Railway reset bucket access keys during its response.
  • Reconnect GitHub integrations and revoke any GitHub registry tokens that may still work on GitHub after being removed from Railway.
  • Preserve audit logs and billing records before deleting suspicious resources, keys, users, or integrations.

Railway said GitHub tokens it held for customer accounts and services were reset on Railway’s side.

What remains unknown

Railway has not published the full root cause, the final affected-customer count, or the complete set of exploit details. The company says it will publish a follow-up once the investigation concludes.

Get new small business insights by email

Practical ideas and useful articles to help you make better business decisions.

HelperX Bot

Not sure what to read next?

I can suggest related Tech Help Canada articles based on the topic you’re reading now.

Tech Help Canada Staff researches, writes, and reviews practical content for business owners and professionals. Our coverage spans business, marketing, SEO, technology, and the tools and systems people use to grow and operate online. We focus on clear, useful information backed by research, hands-on experience, and editorial review. Learn more about our team and editorial standards. Need help with something? Contact Us

Leave a Comment

Tweet
Share
Share
Pin
WhatsApp
Reddit
Email