AWS Infrastructure

ChartChat's AWS infrastructure is managed with Terraform in a separate repository (chartchat-terraform/chartchat-infra). There are three Terraform workspaces — UAT, Production, and Shared — each with its own state.

---

Workspaces

Workspace	Directory	Terraform state key	Purpose
UAT	aws/uat/	uat/terraform.tfstate	UAT environment
Production	aws/prod/	prod/terraform.tfstate	Production environment
Shared	aws/shared/	shared/terraform.tfstate	Shared resources (LLM server)

---

Terraform State

State is stored remotely in S3 so the whole team shares the same state:

Setting	Value
Bucket	chartchat-terraform-state-184995899457-us-east-2-an
Region	us-east-2

Create this bucket manually once before running terraform init for the first time.

---

Region

All resources are deployed in us-east-2 (Ohio).

---

EC2 — App Server

Setting	UAT	Production
Instance type	t3a.large	t3a.large
Root volume	60 GB gp3	AMI default
AMI	Set in terraform.tfvars	Set in terraform.tfvars
Elastic IP	Existing EIP reused	Existing EIP reused
Termination protection	Enabled	Enabled
IAM role	chartchat-uat-ec2-role	chartchat-prod-ec2-role

The EC2 instance runs all services:

• nginx (reverse proxy + static file serving)
• PM2 (Node.js server process manager)
• FastAPI AI tools server
• Jenkins (both environments — port 8080)

Elastic IP

Terraform looks up the existing EIP by public IP address and associates it with the newly created EC2 instance on every terraform apply. The EIP is never destroyed — it is permanent.

AMI lifecycle

The ami field in the EC2 resource has ignore_changes = [ami] — Terraform creates the instance with the specified AMI but never replaces it if the AMI changes. To deploy a new AMI, destroy and recreate the instance.

---

Security Groups

UAT — App Server

Port	Protocol	Source	Purpose
22	TCP	0.0.0.0/0	SSH access
80	TCP	0.0.0.0/0	HTTP
443	TCP	0.0.0.0/0	HTTPS
8000	TCP	0.0.0.0/0	FastAPI AI tools
8103	TCP	0.0.0.0/0	Medplum API server
8080	TCP	0.0.0.0/0	Jenkins

Production — ALB

Port	Protocol	Source	Purpose
80	TCP	0.0.0.0/0	HTTP (redirected to HTTPS by ALB)
443	TCP	0.0.0.0/0	HTTPS (terminated at ALB)

Production — App Server

Port	Protocol	Source	Purpose
22	TCP	0.0.0.0/0	SSH access
80	TCP	ALB security group only	HTTP from ALB
8000	TCP	0.0.0.0/0	FastAPI AI tools
8103	TCP	0.0.0.0/0	Medplum API server
8080	TCP	0.0.0.0/0	Jenkins

warning

On production, ports 8000, 8103, and 8080 are open directly to the internet. Only port 80 (HTTP) is restricted to ALB-only traffic. Consider locking down these ports to known IP ranges if internet-facing exposure is not required.

---

Application Load Balancer (Production only)

Production uses an ALB in front of the EC2 instance for HTTPS termination via ACM.

Setting	Value
Type	Application Load Balancer (internet-facing)
HTTPS cert	ACM certificate ARN set in terraform.tfvars
TLS policy	ELBSecurityPolicy-TLS13-1-2-2021-06
Target group port	80 (forwards to EC2 HTTP)
Health check path	/healthcheck
Health check interval	30s, timeout 10s
Healthy threshold	2 consecutive passes
Unhealthy threshold	3 consecutive fails

HTTP → HTTPS redirect: The HTTP listener (port 80) returns a 301 redirect to HTTPS — no traffic reaches the EC2 on port 80 directly.

---

RDS — PostgreSQL

UAT

Setting	Value
Engine	PostgreSQL 17.6
Instance class	db.t4g.small
Storage	20 GB gp2, auto-scales to 100 GB
Storage encrypted	Yes
Availability zone	us-east-2b
Publicly accessible	Yes (restricted by security group)
Backup retention	7 days
Backup window	03:00–04:00 UTC
Maintenance window	Monday 04:00–05:00 UTC
Deletion protection	Disabled
skip_final_snapshot	true
Restored from snapshot	chartchat-dev-db-snapshot-18-05-2026
apply_immediately	true

Production

Setting	Value
Engine	PostgreSQL 17.6
Instance class	db.t4g.small
Storage	20 GB gp2, auto-scales to 100 GB
Storage encrypted	Yes
Availability zone	us-east-2b
Publicly accessible	No
Backup retention	7 days
Backup window	03:00–04:00 UTC
Maintenance window	Monday 04:00–05:00 UTC
Deletion protection	Enabled
skip_final_snapshot	false (takes final snapshot on deletion)
Final snapshot identifier	chartchat-prod-db-final-snapshot

RDS Security Group

• Both: Allows port 5432 from the app server security group
• UAT only: Also has a public rule (0.0.0.0/0 on port 5432) for direct access — should be removed after setup is complete

Credentials

RDS credentials are stored in AWS Secrets Manager at chartchat/{env}/database as a JSON object:

{
  "host": "<rds-endpoint>",
  "port": 5432,
  "dbname": "chartchat",
  "username": "postgres",
  "password": "<password>"
}

The SSM parameter DatabaseSecrets stores the ARN of this secret. Medplum reads the ARN from SSM and fetches the actual credentials from Secrets Manager at startup.

Connection pool

Both environments set database.maxConnections = 100 in SSM. This is the Node.js pg pool ceiling. The pool dynamically shrinks under low load and only approaches this ceiling under burst traffic. connectionTimeoutMillis ensures requests fail fast if the pool is fully busy.

---

S3 — Binary Storage

Setting	UAT	Production
Bucket name	chartchat-184995899457-us-east-2-an	chartchat-prod-184995899457-us-east-2
Managed by Terraform	No — account regional namespace bucket	Yes
Public access block	Manual	All public access blocked (all 4 flags enabled)
Versioning	Manual	Enabled
Purpose	Medplum binary/file storage (attachments, documents, images)	Same

The bucket name is referenced in IAM policies and SSM parameters (binaryStorage = s3:{bucket-name}). UAT bucket settings must be managed manually via the AWS Console.

S3 versioning on the production bucket means every overwrite or delete of a binary file retains the previous version, enabling point-in-time file recovery directly from the AWS Console or CLI.

---

SSM Parameter Store

All Medplum server configuration is stored under /chartchat/{env}/. The server reads this on startup with the argument aws:us-east-2:/chartchat/{env}/.

Non-sensitive parameters (String)

Parameter	Default value	Description
/chartchat/{env}/baseUrl	placeholder	Medplum server base URL
/chartchat/{env}/appBaseUrl	placeholder	Frontend app URL
/chartchat/{env}/port	8103	Server port
/chartchat/{env}/binaryStorage	s3:{bucket-name}	S3 binary storage path
/chartchat/{env}/storageBaseUrl	placeholder	Public URL for binary storage
/chartchat/{env}/awsRegion	us-east-2	AWS region
/chartchat/{env}/supportEmail	placeholder	Support email address
/chartchat/{env}/adminClientId	placeholder	Medplum admin client ID
/chartchat/{env}/allowedOrigins	placeholder	Comma-separated allowed CORS origins
/chartchat/{env}/vmContextBotsEnabled	true	Enables VM context bots
/chartchat/{env}/defaultBotRuntimeVersion	vmcontext	Bot runtime version
/chartchat/{env}/introspectionEnabled	true	GraphQL introspection
/chartchat/{env}/saveAuditEvents	true	Persist audit events to DB
/chartchat/{env}/logAuditEvents	true	Log audit events
/chartchat/{env}/logRequests	true	Log all HTTP requests
/chartchat/{env}/maxJsonSize	1mb	Max request body size
/chartchat/{env}/maxBatchSize	50mb	Max batch request size
/chartchat/{env}/shutdownTimeoutMilliseconds	30000	Graceful shutdown timeout
/chartchat/{env}/database.maxConnections	100	Node.js pg connection pool ceiling
/chartchat/{env}/DatabaseSecrets	ARN	ARN of the database Secrets Manager secret
/chartchat/{env}/RedisSecrets	ARN	ARN of the Redis Secrets Manager secret

Sensitive parameters (SecureString)

Parameter	Description
/chartchat/{env}/bullBoardPassword	Bull Board queue dashboard password
/chartchat/{env}/googleClientId	Google OAuth client ID
/chartchat/{env}/googleClientSecret	Google OAuth client secret
/chartchat/{env}/recaptchaSiteKey	reCAPTCHA site key
/chartchat/{env}/recaptchaSecretKey	reCAPTCHA secret key
/chartchat/{env}/smtp	SMTP config JSON {"host":"...","port":465,"username":"...","password":"..."}

note

Terraform creates all SSM parameters with "placeholder" values and lifecycle.ignore_changes = [value]. After terraform apply, fill the real values manually in the AWS Console. Terraform will never overwrite them.

---

Secrets Manager

Secret name	Contents	Used by
chartchat/{env}/database	RDS credentials JSON	Medplum server via SSM ARN pointer
chartchat/{env}/redis	Redis credentials JSON {"host":"localhost","port":6379,"password":"..."}	Medplum server via SSM ARN pointer
chartchat/{env}/app-secrets	All application env vars as a single JSON object (EHR credentials, Medplum client config, API keys, redirect URIs, etc.)	App server at startup

Populate all three manually in the AWS Console after terraform apply.

---

IAM

EC2 Instance Role — chartchat-{env}-ec2-role

Attached policies:

Policy	Permissions
chartchat-{env}-s3-full-access	s3:ListBucket, s3:GetObject, s3:PutObject, s3:DeleteObject on the S3 bucket
chartchat-{env}-secrets-manager	SSM GetParameter* on /chartchat/{env}/*, KMS decrypt for SecureStrings, Secrets Manager GetSecretValue for DB, Redis, and app-secrets
CloudWatchAgentServerPolicy	AWS managed policy — allows the CloudWatch Agent to publish memory and disk metrics

DLM Role — chartchat-{env}-dlm-role

Uses the AWS managed policy AWSDataLifecycleManagerServiceRoleForAMIManagement to allow DLM to take AMI snapshots.

---

DLM — Daily AMI Snapshots

Automatically creates a daily AMI of the app server EC2 instance.

Setting	Value
Schedule	Daily at 03:00 UTC
Retention	Last 7 AMIs
Target	EC2 instances tagged Name = chartchat-{env}-server

This means at any time you have 7 daily restore points for the full server (OS, nginx config, app code, Jenkins config).

---

CloudWatch

Both environments have a CloudWatch dashboard and SNS alert topic. Alerts are emailed to the address in alert_email in terraform.tfvars.

Alarms

Alarm	Threshold	UAT	Prod
RDS connections high	> 150 (75% of max 200)	Yes	Yes
RDS CPU high	> 80%	Yes	Yes
RDS freeable memory low	< 100 MB (UAT) / < 200 MB (Prod)	Yes	Yes
RDS free storage low	< 2 GB	Yes	Yes
App server CPU high	> 80%	Yes	Yes
App server status check failed	Any failure	Yes	Yes
App server memory high	> 85%	Yes	Yes
App server disk high	> 80% on /	Yes	Yes
LLM server CPU high	> 90%	No	Yes
LLM server status check failed	Any failure	No	Yes
ALB 5XX errors high	> 10 per minute	No	Yes
ALB unhealthy hosts	> 0	No	Yes

Memory and disk alarms require the CloudWatch Agent to be installed and running on the EC2 instance (pushed via SSM Run Command). Without it, those alarms will have no data.

---

LLM Server (AI Tools)

The LLM server is a GPU EC2 instance (g5.xlarge) shared between UAT and Production. It runs the FastAPI AI tools service and is managed in the aws/shared/ Terraform workspace — completely independent of the UAT and Production workspaces.

Setting	Value
Instance ID	i-06e93d4e5833e8c7e
Instance type	g5.xlarge (GPU instance — 1× NVIDIA A10G)
AMI	ami-072159ab3176afd71 (chartchat-llm-ami-2026-06-08)
Key pair	ChartChat-AI-Server
Subnet	subnet-024cccfad278a3557
Security group	sg-0e3f74e9e584fd681 (managed manually — not in Terraform)
Elastic IP allocation	eipalloc-07d05d765d1a2881f
Root volume	100 GB gp3, delete_on_termination = false
IAM role	None assigned
Terraform state key	shared/terraform.tfstate
ignore_changes	ami, root_block_device, user_data, tags — manage manually

Root volume retention

delete_on_termination = false means if the EC2 instance is ever terminated, the root EBS volume is not deleted automatically. This preserves the model weights and AI tooling data on the volume even if the instance is accidentally terminated.

Elastic IP

The LLM server has a permanent Elastic IP (eipalloc-07d05d765d1a2881f) associated via Terraform. The EIP itself has ignore_changes = [instance] — Terraform manages the association resource separately so the EIP is never accidentally re-assigned.

Importing into Terraform state

If the Terraform state is ever lost, re-import with:

cd aws/shared
terraform import aws_instance.llm_server i-06e93d4e5833e8c7e
terraform import aws_eip.llm_server eipalloc-07d05d765d1a2881f

warning

The LLM instance is managed in aws/shared/ — do not import or recreate it from the uat/ or prod/ workspaces. The environment-level llm-server.tf files are commented out intentionally.

---

Terraform Workflow

First-time setup

cd aws/uat   # or aws/prod
cp terraform.tfvars.example terraform.tfvars
# Fill in terraform.tfvars with real values
terraform init
terraform plan
terraform apply

Outputs

After terraform apply:

Output	Description	UAT	Prod
server_public_ip	Elastic IP of the app server	Yes	Yes
server_instance_id	EC2 instance ID	Yes	Yes
rds_endpoint	RDS connection endpoint	Yes	Yes
alb_dns_name	ALB DNS name — point domain CNAME here	No	Yes

After applying — manual steps

1. Fill real values into all SSM parameters (replace "placeholder")
2. Populate all three Secrets Manager secrets: database, redis, app-secrets
3. Remove the temporary public RDS rule in UAT once direct DB access is no longer needed

---

Environment Differences Summary

Resource	UAT	Production
ALB	No	Yes
HTTPS termination	nginx + Certbot	ALB + ACM
RDS deletion protection	Disabled	Enabled
RDS publicly accessible	Yes (+ public SG rule)	No
RDS final snapshot on delete	No	Yes
S3 bucket	chartchat-184995899457-us-east-2-an (manual)	chartchat-prod-184995899457-us-east-2 (Terraform)
LLM server alarms	No	Yes
ALB alarms	No	Yes
RDS memory alarm threshold	100 MB	200 MB