Modernize Grails 8 content negotiation: framework MIME defaults and honor the Accept header

[codeconsole]

Summary

This modernizes Grails' content-negotiation defaults for Grails 8, in two related parts:

1. MIME types are now framework defaults. New applications no longer need a grails.mime.types block in application.yml; the framework supplies the full set out of the box. A declared grails.mime.types still replaces the defaults (unchanged from Grails 7), with an opt-in flag to merge instead.
2. The Accept header is honored for all clients by default. The legacy "ignore the Accept header for browser user agents" behavior is now opt-in. This fixes browser fetch()/axios calls that request application/json but receive HTML.

Both changes are backward compatible for typical apps and are covered by upgrade notes.

---

Part 1 — MIME types as framework defaults

Previously, every generated app carried a ~15-line grails.mime.types block, while the framework's built-in fallback (MimeType.createDefaults()) only knew 6 types. Removing the block silently shrank content negotiation to those 6.

This change:

• Expands MimeType.createDefaults() to the full set new apps historically declared: all, atom, css, csv, form, html, js, json, multipartForm, pdf, rss, text, hal, xml. This is what's used when no grails.mime.types is configured.
• Keeps the historical contract: a declared grails.mime.types still replaces the defaults (the map you declare is authoritative, and the first entry is the default format). Existing apps — including those that deliberately narrowed the set — are unaffected, and restricting the negotiable types still works exactly as before.
• Adds an opt-in flag, grails.mime.mergeDefaults (default false): when true, a declared grails.mime.types is merged over the defaults instead — an entry overrides the default for its extension, extensions you don't declare keep their defaults, and declared entries are ordered first. This gives the "add one custom type without re-declaring the full list" convenience for those who want it.
• Stops generating the block from Grails Forge (GSP / JSON / Markup view features) and from the web and rest-api profiles.

Result: a new app's application.yml is cleaner, existing configs behave exactly as before, and merging is available on request.

The following default is no longer required to be added to every Grails application.yml:

grails:
    mime:
        types:
            all: '*/*'
            atom: application/atom+xml
            css: text/css
            csv: text/csv
            form: application/x-www-form-urlencoded
            html:
              - text/html
              - application/xhtml+xml
            js: text/javascript
            json:
              - application/json
              - text/json
            multipartForm: multipart/form-data
            pdf: application/pdf
            rss: application/rss+xml
            text: text/plain
            hal:
              - application/hal+json
              - application/hal+xml
            xml:
              - text/xml
              - application/xml

---

Part 2 — Honor the Accept header by default (stop User-Agent sniffing)

What changed

grails.mime.disable.accept.header.userAgents previously defaulted to ['Gecko', 'WebKit', 'Presto', 'Trident'], which caused Grails to ignore the Accept header for any request whose User-Agent matched a major browser engine. It now defaults to unset, so the Accept header is honored for every client. The browser blocklist remains available as an explicit opt-in.

Legacy circa 2007 broken default configuration:

grails:
    mime:
        disable:
            accept:
                header:
                    userAgents:
                        - Gecko
                        - WebKit
                        - Presto
                        - Trident

Why the old default is wrong today

It works around a problem that no longer exists. The blocklist dates to the Firefox 2/3 era, when browsers sent:

text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5

— ranking XML above HTML. Modern browsers send a well-formed header that ranks HTML highest:

Chrome / Firefox / Safari:  text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8

text/html is q=1.0 and application/xml is q=0.9, so ordinary q-value negotiation already resolves browser navigations to HTML. No sniffing required.

It actively breaks modern AJAX. The old code exempted "XHR" requests — but only those carrying X-Requested-With: XMLHttpRequest, a header jQuery adds automatically and fetch() / axios do not. So in a current SPA:

fetch('/api/thing', { headers: { Accept: 'application/json' } })

matched the browser User-Agent, the Accept header was discarded before it was even read, and the response came back as HTML — even though the client explicitly asked for JSON. That is the concrete bug this fixes.

How Spring Boot / Spring MVC handles it

Spring takes the simpler, opposite approach: it trusts the client's explicit signals and never inspects User-Agent.

• Spring MVC's ContentNegotiationManager resolves the representation from the Accept header (and optionally an explicit format parameter). There is no User-Agent strategy.
• A @RestController method with Accept: application/json selects its HttpMessageConverter purely from the Accept header → JSON, regardless of which browser made the request.
• Spring has moved further toward relying only on explicit signals: path-extension content negotiation was disabled by default (favorPathExtension = false) in Spring Framework 5.3+ specifically because deriving the response type from an implicit signal (the URL suffix) enabled Reflected File Download (RFD) attacks.

Grails 8 now aligns with Spring: honor the explicit Accept header; don't branch on an ambient, spoofable one.

Security / robustness considerations of User-Agent-based negotiation

Beyond correctness, negotiating on User-Agent has real downsides:

• Cache confusion / poisoning. When a single URL returns different representations based on User-Agent, a correct implementation must emit Vary: User-Agent. Grails' UA-based negotiation does not, so a shared / CDN cache keyed on the URL can store one representation (say HTML) and serve it to a client that needs another (a JSON API consumer), or the reverse. Honoring Accept removes this class of cache mismatch from the default configuration.
• Spoofable, non-contractual signal. User-Agent is entirely client-controlled. Gating the response type on it makes server behavior depend on a value the client can set arbitrarily — fragile to reason about and harder to cover with security tooling / WAF rules that expect a stable request → representation contract.
• Representation confusion. Returning HTML to a caller that asked for application/json can cause client code to mis-handle the body (parsing an HTML error page as JSON, or rendering untrusted HTML it never expected).

None of these is a critical vulnerability on its own, but together they are concrete reasons the explicit-Accept approach is the safer default.

Dead-code cleanup

While here, removed a sibling dead path — if (msie) header = '*/*' — whose guard (userAgent ==~ /msie(?i)/) never matches a real Internet Explorer User-Agent string and so never executed.

---

Compatibility

• Apps that declare grails.mime.types are fully unaffected — the declared map replaces the defaults exactly as before, so you get precisely the types you declared (the first entry remains the default format). Opt into merging with grails.mime.mergeDefaults: true if you'd rather add to the defaults.
• Browser page loads still resolve to HTML, because modern browsers rank text/html highest — withFormat { html { … } json { … } } keeps serving HTML to browsers and JSON to clients that ask for it.
• To restore the Grails 7 behavior, set grails.mime.disable.accept.header.userAgents: [Gecko, WebKit, Presto, Trident].
• Upgrade notes added (sections 27 and 28 of the Grails 8 upgrade guide).

Testing

• Full suites pass for grails-mimetypes, grails-web-common, and grails-test-suite-web; Grails Forge view specs pass.
• New / updated specs cover: the full default MIME set, replace-by-default and declared ordering, the opt-in grails.mime.mergeDefaults merge/override behavior, the Accept header being honored for non-XHR browser requests, and the opt-back-in config restoring the legacy User-Agent behavior.

[testlens-app]

✅ All tests passed ✅

🏷️ Commit: 0df0959
▶️ Tests: 47710 executed
⚪️ Checks: 46/46 completed

---

Learn more about TestLens at testlens.app.

[sbglasius]

cannot see any flaws in this implementation. And simplifying it for the users is always good.

[bito-code-review]

The documentation changes in this pull request update the content negotiation guide to reflect that Grails 8 now provides built-in default mime types. The guide clarifies that developers no longer need to declare these defaults in application.yml for new applications, and explains that providing a custom grails.mime.types configuration will still replace the entire set of defaults, maintaining backward compatibility with earlier versions.

[jdaugherty]

I don't see the test applications being changed to remove this config by default. Also, we need to add or change one of the test applications so that the merge defaults behavior is tested / shown working in an end to end app.

[codeconsole]

Thanks @jdaugherty — both points addressed.

1. End-to-end mergeDefaults demo (app1)

app1 now declares only a single custom MIME type with merging enabled:

grails:
    mime:
        mergeDefaults: true
        types:
            vnd: application/vnd.app1.v1+json

A new ContentNegotiationController.mergedFormats action plus two ContentNegotiationSpec tests prove the behavior over the wire:

• Accept: application/vnd.app1.v1+json → the declared custom type is negotiated.
• Accept: application/json → still resolves to JSON even though only vnd is declared, because the framework defaults are merged in.

Crucially, app1's entire existing ContentNegotiationSpec (json/xml/html negotiation, respond, extensions, etc.) continues to pass with only vnd declared — that is itself the end-to-end proof that mergeDefaults brings the defaults back in (with mergeDefaults off / replace semantics, those would all fail).

2. Removed the redundant default block from the test apps

Removed the now-redundant grails.mime.types block from 44 test apps that carried the standard web-profile default set (the exact set the framework now supplies). For app3/4/5 and geb-context-path I removed only the types: map and left their explicit disable.accept.header.userAgents opt-in intact.

A few apps were intentionally left as-is because removing the block there is not a no-op:

• JSON-first / REST-ordered apps (e.g. views-functional-tests, issue-views-182, the graphql/* apps, hibernate{5,7}/grails-data-service, mongodb/{test-data-service,gson-templates}, async-events-pubsub-demo): these declare json first, which makes JSON their default response format. The framework default order is all-first (→ HTML), so dropping the block flips the default for a bare respond with no Accept header and breaks those apps' tests. They keep their explicit ordering.
• external-configuration and plugins/exploded: their specs assert grails.mime.types.* config values directly (they exercise config access), so the explicit block is the thing under test.
• Deliberately narrowed sets (micronaut-hibernate{5,7}, spring-dependency-management, the multi-datasource/multitenant hibernate apps, issue-10279, test-phases): these declare a reduced set on purpose; the PR's contract is that a declared map replaces the defaults, so they're unaffected and left untouched.

Verified locally by running the affected integration suites (app1 incl. the new merge tests, views-functional-tests, external-configuration, plugins/exploded, the hibernate tenant apps, and a representative sample of the stripped apps), plus :grails-mimetypes:test.

done