.. _embargoed_hardware_issues:

Embargoed hardware issues
=========================

Scope
-----

Hardware issues which result in security problems are a different category
of security bugs than pure software bugs which only affect the Linux
kernel.

Hardware issues like Meltdown, Spectre, L1TF etc. must be treated
differently because they usually affect all Operating Systems ("OS") and
therefore need coordination across different OS vendors, distributions,
silicon vendors, hardware integrators, and other parties. For some of the
issues, software mitigations can depend on microcode or firmware updates,
which need further coordination.

.. _Contact:

Contact
-------

The Linux kernel hardware security team is separate from the regular Linux
kernel security team.

The team only handles developing fixes for embargoed hardware security
issues. Reports of pure software security bugs in the Linux kernel are not
handled by this team and the reporter will be guided to contact the regular
Linux kernel security team (:ref:`Documentation/admin-guide/
<securitybugs>`) instead.

The team can be contacted by email at <hardware-security@kernel.org>. This
is a private list of security officers who will help you coordinate a fix
according to our documented process.

The list is encrypted and email to the list can be sent by either PGP or
S/MIME encrypted and must be signed with the reporter's PGP key or S/MIME
certificate. The list's PGP key and S/MIME certificate are available from
the following URLs:

  - PGP: https://www.kernel.org/static/files/hardware-security.asc
  - S/MIME: https://www.kernel.org/static/files/hardware-security.crt

While hardware security issues are often handled by the affected silicon
vendor, we welcome contact from researchers or individuals who have
identified a potential hardware flaw.

Hardware security officers
^^^^^^^^^^^^^^^^^^^^^^^^^^

The current team of hardware security officers:

  - Linus Torvalds (Linux Foundation Fellow)
  - Greg Kroah-Hartman (Linux Foundation Fellow)
  - Thomas Gleixner (Linux Foundation Fellow)

Operation of mailing-lists
^^^^^^^^^^^^^^^^^^^^^^^^^^

The encrypted mailing-lists which are used in our process are hosted on
Linux Foundation's IT infrastructure. By providing this service, members
of Linux Foundation's IT operations personnel technically have the
ability to access the embargoed information, but are obliged to
confidentiality by their employment contract. Linux Foundation IT
personnel are also responsible for operating and managing the rest of
kernel.org's infrastructure.

The Linux Foundation's current director of IT Project infrastructure is
Konstantin Ryabitsev.


Non-disclosure agreements
-------------------------

The Linux kernel hardware security team is not a formal body and therefore
unable to enter into any non-disclosure agreements.  The kernel community
is aware of the sensitive nature of such issues and offers a Memorandum of
Understanding instead.


Memorandum of Understanding
---------------------------

The Linux kernel community has a deep understanding of the requirement to
keep hardware security issues under embargo for coordination between
different OS vendors, distributors, silicon vendors, and other parties.

The Linux kernel community has successfully handled hardware security
issues in the past and has the necessary mechanisms in place to allow
community compliant development under embargo restrictions.

The Linux kernel community has a dedicated hardware security team for
initial contact, which oversees the process of handling such issues under
embargo rules.

The hardware security team identifies the developers (domain experts) who
will form the initial response team for a particular issue. The initial
response team can bring in further developers (domain experts) to address
the issue in the best technical way.

All involved developers pledge to adhere to the embargo rules and to keep
the received information confidential. Violation of the pledge will lead to
immediate exclusion from the current issue and removal from all related
mailing lists. In addition, the hardware security team will also exclude
the offender from future issues. The impact of this consequence is a highly
effective deterrent in our community. In case a violation happens the
hardware security team will inform the involved parties immediately. If you
or anyone else becomes aware of a potential violation, please report it
immediately to the Hardware security officers.


Process
^^^^^^^

Due to the globally distributed nature of Linux kernel development,
face-to-face meetings are almost impossible to address hardware security
issues.  Phone conferences are hard to coordinate due to time zones and
other factors and should be only used when absolutely necessary. Encrypted
email has been proven to be the most effective and secure communication
method for these types of issues.

Start of Disclosure
"""""""""""""""""""

Disclosure starts by emailing the Linux kernel hardware security team per
the Contact section above.  This initial contact should contain a
description of the problem and a list of any known affected silicon. If
your organization builds or distributes the affected hardware, we encourage
you to also consider what other hardware could be affected.  The disclosing
party is responsible for contacting the affected silicon vendors in a
timely manner.

The hardware security team will provide an incident-specific encrypted
mailing list which will be used for initial discussion with the reporter,
further disclosure, and coordination of fixes.

The hardware security team will provide the disclosing party a list of
developers (domain experts) who should be informed initially about the
issue after confirming with the developers that they will adhere to this
Memorandum of Understanding and the documented process. These developers
form the initial response team and will be responsible for handling the
issue after initial contact. The hardware security team is supporting the
response team, but is not necessarily involved in the mitigation
development process.

While individual developers might be covered by a non-disclosure agreement
via their employer, they cannot enter individual non-disclosure agreements
in their role as Linux kernel developers. They will, however, agree to
adhere to this documented process and the Memorandum of Understanding.

The disclosing party should provide a list of contacts for all other
entities who have already been, or should be, informed about the issue.
This serves several purposes:

 - The list of disclosed entities allows communication across the
   industry, e.g. other OS vendors, HW vendors, etc.

 - The disclosed entities can be contacted to name experts who should
   participate in the mitigation development.

 - If an expert who is required to handle an issue is employed by a listed
   entity or member of an listed entity, then the response teams can
   request the disclosure of that expert from that entity. This ensures
   that the expert is also part of the entity's response team.

Disclosure
""""""""""

The disclosing party provides detailed information to the initial response
team via the specific encrypted mailing-list.

From our experience, the technical documentation of these issues is usually
a sufficient starting point, and further technical clarification is best
done via email.

Mitigation development
""""""""""""""""""""""

The initial response team sets up an encrypted mailing-list or repurposes
an existing one if appropriate.

Using a mailing list is close to the normal Linux development process and
has been successfully used to develop mitigations for various hardware
security issues in the past.

The mailing list operates in the same way as normal Linux development.
Patches are posted, discussed, and reviewed and if agreed upon, applied to
a non-public git repository which is only accessible to the participating
developers via a secure connection. The repository contains the main
development branch against the mainline kernel and backport branches for
stable kernel versions as necessary.

The initial response team will identify further experts from the Linux
kernel developer community as needed.  Any involved party can suggest
further experts to be included, each of which will be subject to the same
requirements outlined above.

Bringing in experts can happen at any time in the development process and
needs to be handled in a timely manner.

If an expert is employed by or a member of an entity on the disclosure list
provided by the disclosing party, then participation will be requested from
the relevant entity.

If not, then the disclosing party will be informed about the experts'
participation. The experts are covered by the Memorandum of Understanding
and the disclosing party is requested to acknowledge their participation.
In the case where the disclosing party has a compelling reason to object,
any objection must to be raised within five working days and resolved with
the incident team immediately. If the disclosing party does not react
within five working days this is taken as silent acknowledgment.

After the incident team acknowledges or resolves an objection, the expert
is disclosed and brought into the development process.

List participants may not communicate about the issue outside of the
private mailing list. List participants may not use any shared resources
(e.g. employer build farms, CI systems, etc) when working on patches.

Early access
""""""""""""

The patches discussed and developed on the list can neither be distributed
to any individual who is not a member of the response team nor to any other
organization.

To allow the affected silicon vendors to work with their internal teams and
industry partners on testing, validation, and logistics, the following
exception is provided:

	Designated representatives of the affected silicon vendors are
	allowed to hand over the patches at any time to the silicon
	vendor’s response team. The representative must notify the kernel
	response team about the handover. The affected silicon vendor must
	have and maintain their own documented security process for any
	patches shared with their response team that is consistent with
	this policy.

	The silicon vendor’s response team can distribute these patches to
	their industry partners and to their internal teams under the
	silicon vendor’s documented security process. Feedback from the
	industry partners goes back to the silicon vendor and is
	communicated by the silicon vendor to the kernel response team.

	The handover to the silicon vendor’s response team removes any
	responsibility or liability from the kernel response team regarding
	premature disclosure, which happens due to the involvement of the
	silicon vendor’s internal teams or industry partners. The silicon
	vendor guarantees this release of liability by agreeing to this
	process.

Coordinated release
"""""""""""""""""""

The involved parties will negotiate the date and time when the embargo
ends. At that point, the prepared mitigations are published into the
relevant kernel trees.  There is no pre-notification process: the
mitigations are published in public and available to everyone at the same
time.

While we understand that hardware security issues need coordinated embargo
time, the embargo time should be constrained to the minimum time that is
required for all involved parties to develop, test, and prepare their
mitigations. Extending embargo time artificially to meet conference talk
dates or other non-technical reasons creates more work and burden for the
involved developers and response teams as the patches need to be kept up to
date in order to follow the ongoing upstream kernel development, which
might create conflicting changes.

CVE assignment
""""""""""""""

Neither the hardware security team nor the initial response team assign
CVEs, nor are CVEs required for the development process. If CVEs are
provided by the disclosing party they can be used for documentation
purposes.

Process ambassadors
-------------------

For assistance with this process we have established ambassadors in various
organizations, who can answer questions about or provide guidance on the
reporting process and further handling. Ambassadors are not involved in the
disclosure of a particular issue, unless requested by a response team or by
an involved disclosed party. The current ambassadors list:

  ============= ========================================================
  AMD		Tom Lendacky <thomas.lendacky@amd.com>
  Ampere	Darren Hart <darren@os.amperecomputing.com>
  ARM		Catalin Marinas <catalin.marinas@arm.com>
  IBM Power	Madhavan Srinivasan <maddy@linux.ibm.com>
  IBM Z		Christian Borntraeger <borntraeger@de.ibm.com>
  Intel		Tony Luck <tony.luck@intel.com>
  Qualcomm	Trilok Soni <quic_tsoni@quicinc.com>
  RISC-V	Palmer Dabbelt <palmer@dabbelt.com>
  Samsung	Javier González <javier.gonz@samsung.com>

  Microsoft	James Morris <jamorris@linux.microsoft.com>
  Xen		Andrew Cooper <andrew.cooper3@citrix.com>

  Canonical	John Johansen <john.johansen@canonical.com>
  Debian	Ben Hutchings <ben@decadent.org.uk>
  Oracle	Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
  Red Hat	Josh Poimboeuf <jpoimboe@redhat.com>
  SUSE		Jiri Kosina <jkosina@suse.cz>

  Google	Kees Cook <keescook@chromium.org>

  LLVM		Nick Desaulniers <nick.desaulniers+lkml@gmail.com>
  ============= ========================================================

If you want your organization to be added to the ambassadors list, please
contact the hardware security team. The nominated ambassador has to
understand and support our process fully and is ideally well-connected in
the Linux kernel community.

Encrypted mailing-lists
-----------------------

We use encrypted mailing lists for communication. The operating principle
of these lists is that email sent to the list is encrypted either with the
list's PGP key or with the list's S/MIME certificate. The mailing list
software decrypts the email and re-encrypts it individually for each
subscriber with the subscriber's PGP key or S/MIME certificate. Details
about the mailing list software and the setup that is used to ensure the
security of the lists and protection of the data can be found here:
https://korg.wiki.kernel.org/userdoc/remail.

List keys
^^^^^^^^^

For initial contact see the :ref:`Contact` section above. For incident
specific mailing lists, the key and S/MIME certificate are conveyed to the
subscribers by email sent from the specific list.

Subscription to incident-specific lists
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Subscription to incident-specific lists is handled by the response teams.
Disclosed parties who want to participate in the communication send a list
of potential experts to the response team so the response team can validate
subscription requests.

Each subscriber needs to send a subscription request to the response team
by email. The email must be signed with the subscriber's PGP key or S/MIME
certificate. If a PGP key is used, it must be available from a public key
server and is ideally connected to the Linux kernel's PGP web of trust. See
also: https://www.kernel.org/signature.html.

The response team verifies that the subscriber request is valid and adds
the subscriber to the list. After subscription the subscriber will receive
email from the mailing-list which is signed either with the list's PGP key
or the list's S/MIME certificate. The subscriber's email client can extract
the PGP key or the S/MIME certificate from the signature so the subscriber
can send encrypted email to the list.