1555821 - Data collection review for lockwise extension

Status: RESOLVED FIXED

(Lockwise Graveyard :: Security, task)

Description

[Leif Oines [:loines]]

Attachment: Data collection review for data steward

It appears that we haven't had a formal data collection review for this yet. This bug is to do that.

The metrics collected by the extension can be found here:
https://github.com/mozilla-lockwise/lockwise-addon/blob/master/docs/metrics.md#lockbox-desktop-addon-metrics-plan

See attachment for data review request. Gdocs version found here.

Chris, feel free to pass this on to another steward.

Comment 1

[Leif Oines [:loines]]

Comment on attachment 9068836 [details]
Data collection review for data steward

hyperlinks to github didnt convert when pasted as text, so editing them to show up

>Request for data collection review form
>All questions are mandatory. You must receive review from a data steward peer on your responses to these questions before shipping new data collection.
>
>What questions will you answer with this data?
>Are users using Lockbox to retrieve credentials?
>Once downloaded, do users continue to use the app? (i.e., how well are they retained?)
>Does requiring a Firefox Account constitute a roadblock to adoption?
>Why does Mozilla need to answer these questions? Are there benefits for users? Do we need this information to address product or business requirements? Some example responses:
>Telemetry is needed to know whether users are getting value out of the app (retrieving credentials). If they are not, it will guide decision making going fowrard re: product improvements.
>What alternative methods did you consider to answer these questions? Why were they not sufficient?
>No alternative methods were considered because there is no other way of gathering this information except through telemetry.
>Can current instrumentation answer these questions?
>No, this is a new feature and it does not yet have any telemetry associated with it.
>List all proposed measurements and indicate the category of data collection for each measurement, using the Firefox data collection categories on the Mozilla wiki.
>See https://github.com/mozilla-lockwise/lockwise-addon/blob/master/docs/metrics.md#scalar-metrics and https://github.com/mozilla-lockwise/lockwise-addon/blob/master/docs/metrics.md#list-of-planned-metrics-events of metrics.md document. All of these metrics are category 1 and 2. Note that the FxA uid IS NOT collected via the lockwise extension at this point.
>
>Note that the data steward reviewing your request will characterize your data collection based on the highest (and most sensitive) category.
>How long will this data be collected? Choose one of the following:
>I want to permanently monitor this data. 
>Contact: [e-mail], [e-mail]
>What populations will you measure (Note that this is an opt-in extension, not native firefox code so these questions are only semi-applicable).
>Which release channels? All
>Which countries? All
>Which locales? All
>No other filters
>If this data collection is default on, what is the opt-out mechanism for users?
>User can opt out of this telemetry by turning telemetry off.
>Please provide a general description of how you will analyze this data.
>We plan to calculate, among other things:
>The average rate with which a user retrieves a credential or reveals a password
>The distribution of above rates across all users
>The number of times a user: 
>Display the credential list
>Tap a credential in the credential list
>Copy a credential to the clipboard
>Reveal a password
>Autofill a credential stored in Lockbox into another app
>Tap the URI associated with a credential (to open it in an app or browser)
>Where do you intend to share the results of your analysis?
>Only with Mozilla employees/NDA mozillians
>Is there a third-party tool (i.e. not Telemetry) that you are proposing to use for this data collection? 
>Not currently, but events may eventually be sent to amplitude in the medium-to-long term future.
>

Comment 2

[Chris H-C :chutten|PTO (back Oct 23)]

Comment on attachment 9068836 [details]
Data collection review for data steward

DATA COLLECTION REVIEW RESPONSE:

    Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes. Though this collection is Telemetry, it is runtime telemetry so will not be visible in a definitions file or the Probe Dictionary. Instead it is documented in the lockwise repository itself here: https://github.com/mozilla-lockwise/lockwise-addon/blob/master/docs/metrics.md

    Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. This collection is Telemetry so can be controlled through Firefox's Preferences. Additionally, disabling, removing, or just not installing the Lockwise addon will result in none of this data being collected.

    If the request is for permanent data collection, is there someone who will monitor the data over time?

Yes, Leif Oines and Matthew Miller are responsible.

    Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2, Interaction. The bulk of the collection are interaction events within the addon itself.

    Is the data collection request for default-on or default-off?

Default on for all channels, so long as the user installs the addon.

    Does the instrumentation include the addition of any new identifiers?

Unclear. ** Needs Clarification **

    Is the data collection covered by the existing Firefox privacy notice?

Yes.

    Does there need to be a check-in in the future to determine whether to renew the data?

No. This collection is permanent.

---
Result: datareview+, pending clarification.

:loines, could you expand on what an "Item GUID" is in this instance? Is it something generated on the client? What is its domain (is it for logins? Is it to uniquely id them across the system or just within the client? Is it correlated against some other list?)?

Comment 3

[Leif Oines [:loines]]

EDIT: I had this wrong actually . The item guid is randomly generated and used to identify each lockwise entry (login + pw) that is stored. These guids ARE CONSTANT ACROSS PLATFORMS, and could appear in lockwise ios/android telemetry, if the items are accessed there. There is no other "master list" where these guids are avaliable to mozilla, however.

Comment 4

[Chris H-C :chutten|PTO (back Oct 23)]

(In reply to Leif Oines [:loines] from comment #3)

EDIT: I had this wrong actually . The item guid is randomly generated and used to identify each lockwise entry (login + pw) that is stored. These guids ARE CONSTANT ACROSS PLATFORMS, and could appear in lockwise ios/android telemetry, if the items are accessed there. There is no other "master list" where these guids are avaliable to mozilla, however.

So they are generated on the initial client and are sync'd across platforms? Would that allow us to learn which client_ids are using the same fxa account? (if the same item GUID is used by multiple client_ids, for instance).

Is this part of the collection required in order to answer the data collection review's questions? Could we instead use a different (perhaps client-local) id for the items?

Comment 5

[Leif Oines [:loines]]

(In reply to Chris H-C :chutten from comment #4)

(In reply to Leif Oines [:loines] from comment #3)
So they are generated on the initial client and are sync'd across platforms?

Yes, this is correct, but they can only be generated on desktop (the mobile apps cannot modify credentials).

Would that allow us to learn which client_ids are using the same fxa account? (if the same item GUID is used by multiple client_ids, for instance).

If the user modifies/accesses the same item on multiple desktop profiles, it could be possible to link them, but an event involving that item has to be generated on each client first. One of the product questions we care most about is cross-device (mostly mobile + desktop) access of the same credentials. Note that the mobile lockwise apps are stand-alone and have unique client_ids (i.e. are not the same as firefox android/ firefox ios). For the mobile apps, I believe we have legal approval to to do this sort of cross-device analysis. Sandy can you confirm this, I think it was michael feldman?

However I get that we are in desktop realm now so things may be different.

Is this part of the collection required in order to answer the data collection review's questions? Could we instead use a different (perhaps client-local) id for the items?

See above, yes product management cares very much about cross-device access. Sorry Chris, I could have been more explicit about this to begin with.

Comment 6

[Chris H-C :chutten|PTO (back Oct 23)]

No worries, it's very clear now. I'm a little concerned about sending the Telemetry client_id along with these item GUIDs because it gives us the capability to cross-reference profiles in a way that we previously didn't have.

We have come across this sort of thing before with FxA. The way we handled it was through a separate "sync" ping that doesn't submit a client_id (or much of the Environment). Unfortunately the tooling isn't the best for exposing this to addons in an ergonomic way (keywords "multistore" "dynamic scalars" "dynamic events"). You'd have to store your data yourself (ie, not use Telemetry Events or Scalars, but write your own), but you could use browser.telemetry.submitPing to hand it off to Firefox Telemetry to make sure it's sent.

It's either that or we get confirmation that it's okay to cross the streams in this way and I'm just blowing things out of proportion.

Comment 7

[Sandy Sage]

For additional context on the product questions, we've outlined our hypotheses and associated success criteria in the PRD: Lockbox WebExtension: https://docs.google.com/document/d/1gcBQk4yXUjBv0kOeeCM0q4dkzlUt0ZkGzmz30YFaOe0/edit#

Cross-device access is the underling value proposition we're promoting with Lockwise. These metrics inform the core of our product strategy.

Comment 8

[Leif Oines [:loines]]

Hi Alicia, would you mind throwing your hat into the ring here?

A bit of background to catch you up:
The Lockwise mobile apps and desktop extension generate random ids for each entry in the user's password manager. These IDs are not derived from the usernames or passwords, they are random (i.e. they contain no pii). The IDs are also constant (via sync) across all of the Lockwise apps that are connected to an individual Firefox/Sync Account (the desktop extension, android and ios apps). The IDs are also included in the telemetry events that are recorded when a user modifies or accesses a credential.

Chris has raised a valid issue here in that the Lockwise desktop extension telemetry is sent with the desktop client_id, which is also associated with most of the rest of a client's telemetry measurements. If a user of Lockwise:

1. Uses the lockwise extension on two (or more) desktop profiles,
2. Has both profiles connected to sync (FxA)
3. and accesses the same item_id on both of them

Then it would be possible to link telemetry from the two client_ids together via the item id. The concern is that there is a precedent for not allowing this type of cross-client_id correlation (see Chris's description of the sync ping above).

However from a product management perspective, a critical question is whether or not a user is accessing the same items on both their desktop and mobile apps, since an important use case is cross-device access of credentials. Having these item IDs in our telemetry would allow us to answer this. If they are not included, we will not be able to answer it.

We're trying to determine if the use case (in terms of product development) for including the item ids outweighs the possibility of being able to potentially correlate desktop client_ids for users who meet the conditions 1-3 above, or if we should remove these item ids from the lockwise extension metrics.

(note that the lockwise mobile apps are stand-alone and have client_ids that are specific to the app and are not associated e.g. with mobile browser telemetry)

Thank you and your sage counsel is much appreciated

Comment 9

[Georg Fritzsche [:gfritzsche]]

Moving this to the Lockwise bugzilla product.
If this is not a fitting component, maybe Firefox::General is a better fit.

Comment 10

[Alicia Gray]

Hi Leif,

Reviewing...

Comment 11

[Alicia Gray]

Hi Leif,

We strongly advise against connecting to the client_id. As :chutten pointed out in comment 6 we've purposely avoided doing this and we don't want to cross the streams in this manner. It would be best to find an alternative method. Perhaps the option :chutten suggested would be viable for the need.

Please let me know if there are more questions or if we can help with another review of a proposal.

Thanks,
Alicia

Comment 12

[Leif Oines [:loines]]

Thanks Alicia,

We have tentatively decided to remove the item_id from the lockwise extension telemetry before its telemetry is re-enabled. Issue filed here https://github.com/mozilla-lockwise/lockwise-addon/issues/313

Comment 13

[Matthew N. [:MattN]]

Can this be marked as resolved now?