The Department of Defense (DoD) and Major Service policy on the use of commercial services for conducting surveys is provided to help evaluate courses of action necessary due to the reduction in access to these services. In general, DoD policies do not allow the use of commercial services and/or tools for survey-related purposes. Organizations have been established within the Services for development and management, cost estimating, development of survey tools, survey distribution and information collection, analysis of results, and archival of results. However, these may not provide the breadth of fully integrated capabilities and tools found within commercial services. To that end, policies do provide a process for requesting exceptions or waivers when mission operations would be negatively impacted. Policies are summarized, and examples of DoD survey services and contracts for commercial survey services and tools are provided.

1.0  TI Response

The U.S. Army Research and Analysis Center (TRAC) uses Verint software to develop surveys. TRAC manages their software license through the U.S. Army Training and Doctrine Command (TRADOC). TRADOC recently significantly reduced the number of Verint accounts available for use by TRAC. As a result, TRAC lacks enough Verint accounts to meet the demand for development, deployment, and analytics of surveys for the organization. It is imperative that TRAC evaluates possible courses of action to execute future surveys to support studies effectively.

1.1  DoD Guidance on Surveys

The overarching guidance for use of surveys within the DoD is provided by DoD Instruction (DoDI) 8910.01, “Information Collection and Reporting,” [1] and DoDI 1100.13, “DoD Surveys” [2]. DoDI 1100.13 establishes the policies, assigns responsibilities, and provides procedures for information collections involving the use of surveys and implements these policies in accordance with guidance in DoD Manual (DoDM) 8910.01, Volume I, “DoD Information Collections Manual: Procedures for DoD Internal Information Collections,” [3] and Volume 2, “DoD Information Collections Manual: Procedures for DoD Public Information Collections” [4].

DoDI 1100.13 applies to the Office of the Secretary of Defense (OSD), the Military Departments, the Office of the Chairman of the Joint Chiefs of Staff and the Joint Staff, the Combatant Commands, the Inspector General of the Department of Defense, the Defense Agencies, the DoD Field Activities, and all other organizational entities within the DoD. DoDI 1100.13 is applicable to any of these entities that wish to issue surveys that require the following:

• Participation of personnel from more than one DoD or OSD Component, including those of Military Service members; Service members’ spouses; family members eligible for DoD benefits; DoD civilians; Service academy students; and survivors of deceased Service members.
• Participation of DoD personnel that are requested by an external organization, including State, local, or tribal governments, other federal agencies, or public organizations or individuals, or are called for by law with no direct response to Congress required.
• Surveys of members of the public (including government contractors) conducted by any DoD or OSD Component. Public information collections, including surveys of this type must be developed and processed in accordance with the guidelines in DoD Manual 8910.01, Volume 2, “DoD Information Collections Manual: Procedures for DoD Public Information Collections” [2].

DoDI 1100.13 does NOT apply to the following:

• DoD or OSD Components conducting surveys of personnel from only one DoD or OSD Component.
• The Office of the Inspector General of the DoD in fulfilling its statutory duties and functions [2].

Note that the Services and other Components within the DoD may have additional survey approval policies, processes, and procedures.

1.1.1  U.S. Air Force Guidance

Air Force Manual (AFMAN) 36-2664, “Personnel Assessment Program,” states that the Military Force Management Directorate (AF/A1P) establishes policy for the Air Force Survey Program. The Air Force Personnel Center, Air Force Survey Office (AFPC/DSYS) manages the Air Force Survey Program and is the approving and controlling authority for attitude and opinion research. The AFPC/DSYS also assists other Air Force organizations in survey administration, processes, and analysis. AFMAN 36-2664 sections 5 and 6 cover Air Force survey roles and responsibilities, development guidelines, request procedures, process, release of findings, etc. [5].

1.1.2  U.S. Army Guidance

Army Regulation (AR) 25-98, “Information Management Control Requirements Program” covers Army public generic surveys in chapters 5−9 and Army internal surveys in section 6. AR 25-98 states that the Administrative Assistant to the Secretary of the Army (AASA) is responsible for Army-wide control of information management and reporting requirements; however, AASA may designate internal survey-approval authority to other Army activities as needed (AR 25-98, sections 1−9 and 1-9.10.). AR 25-98, section 2-3, Subsection b, Licensure, further states that “[A]ll surveys administered by an Army organization will be submitted for a formal review and determination.”  However, AR 25-98 sections 3-1; 3-2. General; and 3-3. Army Internal Exemptions, state that Army nonsurvey Internal Information Collections (collection of information from Army personnel only) may be exempt from formal review, if they “align with the multi-Service collections exemptions outlined in DoDM 8910.01, Volume 1, Enclosure 3.” AR 25-98, section 4-3, Multi-Service Exemptions, subsection c further states that “Army organizations will notify DA [HQDA] of nonsurvey collections exempt from formal review” [6].

1.1.3  U.S. Navy Guidance

The Navy Personnel Command states that the Navy Survey Program Office administers Navy survey policy per OPNAVINST 5300.8C, “Coordination and Control of Personnel Surveys.” OPNAVINST 5300.8C notes that it is issued in amplification of DoD 8910.1-M, Management and Control of Information Requirements, 30 Jun 98; DoDI 1100.13, Surveys of DoD Personnel, 21 Nov 96; and SECNAV M-5214.1, 1 Dec 2005. OPNAVINST 5300.8C also states that it delineates “policy and procedures for the management, control, review and approval and licensing of Navy military and civilian surveys.”

Section 4. Applicability, states that the instruction applies to “all surveys of Department of Navy (DON) military members and their families and Navy Civilian personnel. Marine Corps surveys are administered under authority of the Commandant, United States Marine Corps.”

Section 8. Responsibilities, states that “The Chief of Naval Operations, Modeling and Analysis Branch, CNO (NI04), or other designee in consultation with the Director, NPRST shall appoint the Navy Survey Approval Manager” and that the Navy Survey Approval Manager will “provide control and coordination of all personnel surveys conducted within the Navy.” Section 8.a.(12)b further notes that the Navy’s Modeling and Analysis Branch (CNO (NI04)) will provide resource management and oversight for the Navy Survey Approval Manager [7, 8].

OPNAVINST 5300.8C covers both active duty and reserve Navy sailors and their families, civilian employees, and those retired from active duty, reserve, or civilian status. The instruction does not apply to surveys of family members of retirees, government contractors, or members of the general public. Surveys of these populations require Office of Management and Budget (OMB) approval. Section 7.a on exemptions notes that the following are exempt from provisions of the instruction:

• Navy Occupational Task Analysis Program (NOTAP, Navy Manpower Analysis Center [NAVMAC]) surveys and related job, task, and competency analysis efforts.
• Navy Training Importance Surveys (TIS, NAVMAC).
• Navy Training Evaluation Surveys (i.e., those that determine the effectiveness of a specific training course among students who took the course).
• Surveys conducted by bases or commands (e.g., Navy Personnel Command) when executed entirely within the base or command (e.g., command equal opportunity surveys, base or command Morale, Welfare, and Recreation [MWR] surveys, etc.).
• Surveys conducted by the Naval Inspector General or other Navy or DoD investigative agencies when the surveys are part of their recognized and official duties.
• Course evaluation surveys, provided the survey deals specifically with content provided in or used to provide the course in question and/or conducted while the course is in session.
• On-site/point-of-service, customer, and patient satisfaction surveys.

OPNAVINST 5300.8C also provides information on Navy internal processes and organizational responsibilities for development, dissemination, analysis, and archiving of data related to surveys including web-based surveys conducted over the Naval enterprise internet using resources such as the Navy Knowledge Online site. OPNAVINST 5300.8C does not discuss or note approval for use of any commercially available tools or services. However, it does note use of the Navy’s Personnel Command Survey Policy website, https://www.public.navy.mil/bupers-npc/organization/bupers/SurveyPolicy/Pages/default.aspx, for requesting approval for surveys. This site states that “[T]he Navy Survey Program Office has access to a variety of tools that can help with data collection efforts and are currently the only set of approved tools for use” [8, 9].

1.2  Survey Services and Tools

AR 25-98, section 1-8.i states that “[C]ommercial platforms, unless specifically approved by CIO/G–6, are not authorized for Army conducted surveys. See chapter 6 for authorized and preferred online survey development sites.” Section 6-2. Policy, section 6-2.d further states that “[C]ommercial platforms, unless specifically approved by CIO/G–6, are not authorized forums for conducting Army internal surveys.” AR 25-98, section 6-2, Policy, section 6-2.c notes that Army use of online survey tools is allowed and provides milSurvey and MAX Survey as examples. The Army’s preferred online survey tool is milSurvey [7].

The following are some examples of U.S. Government-developed and commercially developed Government-approved survey services and tools. Note that this is not a complete list.

1.2.1 milSuite milSurvey

The purpose of milSuite (https://www.milsuite.mil/) is to provide a collection of social business tools for DoD personnel to facilitate professional networking, learning, and innovation through knowledge sharing and collaboration. Part of the milSuite services include milSurvey, a secure online survey platform for the DoD that allows members of the milSuite community to create custom surveys for enterprise distribution with advanced survey statistics to capture, review, and share. Surveys can include everything from simple questionnaires to surveys with conditional branching questions and can provide basic statistical analysis of survey results.

A walk-through wizard and helpful tips are provided to get users started; the milSuite Classroom offers classes on introduction to the milSurvey application, a walk-through for creating and configuring a survey, and management of visibility options for survey participants and an anonymity feature. The milSurvey support forum is available for users to ask questions and connect with the milSurvey team. Note that the tool is not approved for non-work-related Personally Identifiable Information (PII) [9].

1.2.2 MAX Survey

MAX.gov (https://www.max.gov) is a government-wide suite of advanced collaboration, information-sharing, data-collection, publishing, business intelligence, and authentication tools and services used to facilitate cross-government collaboration and knowledge management. MAX.gov tools include MAX Community, MAX Collect, MAX Survey, MAX A-11, MAX Analytics, MAX Authentication, and others. MAX Survey has robust options for navigation and in-survey conditional logic and can be run either with a generic survey link, or, using “token-based” access control with a predefined, precise list of respondents, and each respondent has a unique link to access the surveys. Token-based access control enables tracking of responses and personalized follow-up emails to those who have not responded; this tracking greatly increases response rates.

The surveys can be designed to remain anonymous (by not including the personal information in the result set) or can be set up to automatically provide attribution for surveys where knowledge of the submitter is appropriate. Survey results can be downloaded to Excel or accessed via MAX Analytics. MAX Analytics can then be dynamically displayed within the MAX Community within integrated dashboards [10].

1.2.3 Navy Personnel Command Survey Program Office

The Navy Personnel Command Survey Program Office site (https://www.public.navy.mil/bupers-npc/organization/bupers/SurveyPolicy/Pages/ default.aspx) provides access to examples/templates to request survey approval and tools for estimating costs to complete a survey project as well as costs for respondents to answer the survey, survey FAQs, and the survey archive. The site further notes that

[B]efore selecting any commercially available survey tool, you should check to make sure that it meets all relevant DoD and Navy Information Assurance and IT security requirements (e.g., FedRAMP). Since the data will be stored on commercial servers there is increased risk of harm or embarrassment if the data are somehow compromised. The Navy Survey Program Office has access to a variety of tools that can help with data collection efforts and are currently the only set of approved tools for use. These include online surveys, live polling, and electronic focus group tools [8].

1.2.4 Office of Personnel Management (OPM), Survey Services

OPM psychologists are experts in survey methodology and evaluation and have the unique capability of providing Government-wide benchmark data in addition to private-sector comparisons. The OPM survey administration services include survey design, sampling, communications, data management, statistical analysis, and results reporting. Agencies can have a custom survey developed to meet their specific needs or they can use an OPM standard survey. These standard surveys provide agencies with research-based assessments, streamlined administrations, and benchmarking capability.

Outside of these standard services, OPM offers highly customized services to include multiple data collection methods to assess specific needs such as training needs analysis, action planning, mentoring, and focus groups. Focus groups are used as research tools for gathering in-depth, qualitative information from a group of people in a focused discussion to find out how people feel or think about an issue, idea, product, or service [11].

1.2.5 Government Services Administration (GSA) Federal-Compatible Terms of Service (TOS) Agreements for Official Government Use

The GSA works with agencies to negotiate amended TOS agreements for free, third-party applications, which include steps like removing advertisements so as not to look like endorsements. See OMB Memorandum M–13–10 for more information. Over 70 third-party applications/services have active GSA TOS agreements. A few of these applications, such as IdeaScale, Qualtrics, Quora, Survey Analytics, SurveyMonkey, and Yelp, are related to collecting customer feedback and other survey information. However, this fact does not mean a particular application/service has been approved for use by any specific Agency or Department or Component thereof [12].

Note: The GSA TOS program is no longer active. While individual TOS agreements for applications may still be active, new agreements cannot be added. The full list of free tools and services that have federal-compatible TOS active and inactive agreements can be found on Digital.gov, https://digital.gov/resources/negotiated-terms-of-service-agreements/ [13].

1.2.6 SurveyMonkey

Anecdotally, from DSIAC’s perspective, SurveyMonkey (https://www.surveymonkey.com/) seems to be one of the more popular commercial survey tools. SurveyMonkey has a suite of tools such as SurveyMonkey (online survey platform), SurveyMonkey Enterprise (online enterprise-level survey platform), Audience (surveys of global consumer panels), and others. SurveyMonkey is a GSA-approved Federal Government survey service provider and is also a cloud service provider available through DigitalGov.gov, https://digital.gov/, the Government’s platform for helping public agencies access and procure cloud services. SurveyMonkey has an active GSA TOS agreement in place [13, 14].

1.3  Verint Systems, Inc. EFM Software

EFM (formerly VOVICI Survey Workbench) is produced by Verint Systems, Inc. and is a system of processes and software for centrally managing deployment of surveys, analyzing results, and distributing the results throughout an organization. EFM supports organizations in systematically collecting, analyzing, consolidating, and then using all sources of feedback to improve business processes. Modern EFM solutions can track feedback from customers, employees, vendors, partners, market research, and social media. In addition to aggregating multisource feedback from multiple stakeholders across multiple channels, EFM supports real-time dissemination of actionable information across the organization to where it is most relevant. EFM then enforces accountability for follow-up and service-lapse recovery. The EFM vendors include Confirmit, CXGroup, Clarabridge, InMoment, Intouch Insight, MaritzCX, Medallia, Mopinion, NICE Systems, Netigate, Qualtrics, QuestBack, SurveyMonkey, Verint, and Wootric [15, 16].

Based on the Digital.gov list of active and inactive Federal TOS agreements, Verint does not have a Federal TOS Agreement [17]. However, a quick search of the U.S. Federal System for Award Management (SAM), https://sam.gov/SAM/, formerly Fed Biz Ops (FBO), indicates that Verint has existing Federal Government contracts, including those for the DoD. For example, the Army TRADOC awarded Verint a contract for use of their EFM software product and software maintenance, advice line support, and training services. This contract was a firm-fixed-price delivery order under the GSA Federal Supply Schedule (FSS) Program with a period of performance from 1 June 2020 to 31 May 2024 [18].

This contract did require the Army to issue a Limited Sources Justification per FAR 8.405-6 that was approved, which may prove useful for other federal agencies to justify use of Verint’s EFM software and services. The justification notes that

Verint provides a capability to operate in an approved Federal Risk and Authorization Management Program (FedRAMP) container within the TRADOC Microsoft Azure cloud at impact level 5. Verint is widely used by US ARMY Centers of Excellence (COE). This is also the same software tool used by the Army to provide installation surveys. Other Survey tools investigated lack the depth or breadth of functionality and ensure controls throughout the survey lifecycle. Additionally, most of these survey tools are not designed to run in the GovCloud. While many survey tools operate in the commercial cloud space, they do not provide the security of the survey or results, nor ensure the government ownership of the survey or the result.

In addition,

[A]dopting an alternative software solution that operates in the FedRAMP approved environment, [and is] secure, protects generated data, [and] is highly customizable/scalable and expandable across the TRADOC enterprise, will require a significant investment of money and resources that are presently unavailable. TRADOC previously used Verint EFM survey software at multiple sites to meet their requirements. The continuation of Verint software streamlines the analyses, saves the government manpower hours and maintains continuity within the user community. Consolidation of the requirement saves the command and the Army resources, through contract consolidation and enabling the capability in the TRADOC Cloud environment. Migration to a different survey software would not be able to fully leverage the significant investments already made in previous purchases or permit utilization of previous surveys conducted. Replacing existing products and licenses would be a substantial duplication of cost to the Government, would require extensive data migration to a new environment and would require the generation of new surveys at every COE/course [19].

Various federal solicitation and contract-award repositories including the Defense & Aerospace Competitive Intelligence Service (DACIS), https://www.dacis.com/; Federal Procurement Data System – Next Generation (FPDS-NG), https://www.fpds.gov/fpdsng_cms/index.php/en/; and Loren Data’s SAM Daily™, https://www.fbodaily.com/, were searched for DoD open solicitations and/or contract awards for Verint’s EFM software and/or services since the start of 2019. Results indicated that at least five federal contracts were awarded, and one federal solicitation was open for purchase of Verint EFM software and/or services in the January 2019 through September 2020 time period. Three contracts were awarded by the Army, one by the Coast Guard, and one by OPM. The Tri-Service Medical Education and Training Campus located in Ft. Sam Houston, TX, currently has a solicitation out to purchase Verint EFM software. DoD solicitations and contract awards for Verint EFM services and software are listed in Table 1. Other awards related to Verint were also found, but the information was insufficient to determine if these awards were specifically for EFM services or software.

* NAICS (North American Industry Classification System): The NAICS is the standard used by federal statistical agencies in classifying business establishments for the purpose of collecting, analyzing, and publishing statistical data related to the U.S. business economy. A listing of the 2017 NAICS can be found at https://www.census.gov/cgi-bin/sssd/naics/naicsrch?chart=2017.

** PSC (Product Service Code):The PSCs describe the products, services, and research and development items purchased by the Federal Government and are used in collecting and analyzing federal procurements. A listing of PSCs is on the Acquisition.Gov website, https://www.acquisition.gov/psc-manual.

References

[1] DoD Instruction 8910.01. “Information Collection and Reporting.” https://www.esd.whs.mil/portals/54/documents/dd/issuances/dodi/891001p.pdf, 19 May 2014, Incorporating Change 1 effective 8 July 2020.

[2] DoD Instruction 1100.13. “DoD Surveys.” https://www.esd.whs.mil/Portals/54/Documents/DD/issuances/dodi/110013p.pdf, 15 January 2015, Incorporating Change 1 effective 31 March 2017.

[3] DoD Manual 8910.01, Volume 1. “DoD Information Collections Manual: Procedures for DoD Internal Information Collections.” https://www.esd.whs.mil/Portals/54/Documents/DD/issuances/dodm/891001m_vol1.pdf, 30 June 2014, Incorporating Change 2 effective 19 April 2017.

[4] DoD Manual 8910.01, Volume 2. “DoD Information Collections Manual: Procedures for DoD Public Information Collections.” https://www.esd.whs.mil/Portals/54/Documents/DD/issuances/dodm/891001m_vol2.pdf., 30 June 2014, Incorporating Change 2 effective 19 April 2017.

[5] AFMAN 36-2664. “Personnel Assessment Program.” https://static.e-publishing.af.mil/production/1/af_a1/publication/afman36-2664/afman36-2664.pdf, 16 May 2019.

[6] Army Regulation 25-98. “Information Management Control Requirements Program.” https://armypubs.army.mil/epubs/DR_pubs/DR_a/pdf/web/ARN20910_R25_98_ADMIN_FINAL.pdf, 8 February 2019.

[7] Survey Policy. Navy Personnel Command. https://www.public.navy.mil/bupers-npc/organization/bupers/SurveyPolicy/Pages/default.aspx.

[8] Chief of Naval Operations. “OPNAV INSTRUCTION 5300.83: Coordination and Control of Personnel Surveys.” https://www.secnav.navy.mil/doni/Directives/05000%20General%20Management%20Security%20and%20Safety%20Services/05-300%20Manpower%20Personnel%20Support/5300.8C.pdf, 24 April 2008.

[9] milSuite: milSurvey. U.S. Army PEO-Command, Control, Communications-Tactical, at https://www.milsuite.mil/survey/, accessed 11 September 2020.

[10] MAX.gov Homepage. U.S. Federal Government. https://portal.max.gov/portal/home, accessed 11 September 2020.

[11] Services for Agencies: Assessment & Evaluation. U.S. Office of Personnel Management. https://www.opm.gov/services-for-agencies/assessment-evaluation/surveys-and-related-services/, accessed 11 September 2020.

[12] GSABLOG: Facts on the Federal-Compatible Terms of Service Agreement for Yelp. U.S. General Services Administration., https://www.gsa.gov/blog/2015/08/20/facts-on-the-federal-compatible-terms-of-service-agreement-for-yelp, accessed 11 September 2020.

[13] Negotiated Terms of Service Agreements: A list of free tools that have federal-compatible terms of service agreements. Digital.gov. https://digital.gov/resources/negotiated-terms-of-service-agreements/, accessed 11 September 2020.

[14] Loh, S. (n.d.). SurveyMonkey is now federal government approved. SurveyMonkey. https://www.surveymonkey.com/curiosity/surveymonkey-for-government/, accessed 10 September 2020.

[15] Hanks, R. Are You Ready for Enterprise Feedback Management? Mindshare Technologies. https://web.archive.org/web/20140311164025/http://www.mshare.net/measurement/are_you_ready_for_enterprise_feedback_management.php, accessed 14 September 2020.

[16] Enterprise Feedback Management Report. DMG Consulting LLC. https://www.dmgconsult.com/research-reports-enterprise-feedback-management, accessed 14 September 2020.

[17] Agency Points of Contact for Federal Compatible Terms of Service Agreements. Digital.gov. https://digital.gov/resources/agency-points-of-contact-for-federal-compatible-terms-of-service-agreements/, accessed 14 September 2020.

[18] Verint Enterprise Feedback Mgmt. Software, SW Maint., Advice Line Sup. & Trng. SAM.Gov. https://beta.sam.gov/opp/4c35d282fcb64f31be42838e2e2a3850/view, accessed 14 September 2020.

[19] Justification Review Document FAR 8.405-6 Limited Sources Justification Under the General Services Administration Federal Supply Program, Control No: Eustis 2020-7. U.S. Army Training and Doctrine Command. https://beta.sam.gov/api/prod/opps/v3/opportunities/resources/files/24c37096877d4729997                                                                             e079685055f0b/download?api_ key=null&status=archived&token=, accessed 14 September 2020.