### Redundant display of data on plots

In this series of blog posts, we will use a survey by the U.S. law firm Winston & Strawn to learn about survey methodology. In 2013 the firm produced a 33-page report based on the survey results entitled “The Winston & Strawn International Business Risk Survey 2013”.  To download a PDF of the report, click here.

The plot in the image below comes from page 18 of the report. The survey had asked respondents the question stated in the header, given them eight choices, and this plot presents the results as a graph. Here we will focus on one aspect of that plot: how effectively it presents the sum of the number of times respondents selected each of the risk choices.

Notice that the plot identifies the number of companies selecting a risk by three methods. One is the horizontal x-axis that ranges from zero on the left to 80 on the right. For example, “Rogue employees” is just to the left of the 50 marker on the x-axis so a reader could estimate 47-49 respondents chose if from that bar’s end point, where it reaches on the x-axis, and the figure from the y-axis.

The second method is the numeric label at the end of each bar. “Rogue employees” proclaims a large “48”.

Third, from the bottom axis light, vertical dotted lines extend upward from each interval. These vertical “grid lines” as they are referred to by data visualizers, are spaced at even intervals of five. If there were a label that explained the intervals someone could count nine of them from the left plus a little bit and estimate that 47-49 respondents chose “Rogue Employees”.

The plot would be less cluttered, less redundant, and more precise if it omitted the superfluous grid lines as well as the unnecessary x-axis. It would leave the numeric labels as the salient, immediately understandable statements of the results.

The plot below recreates the original plot using the R programming language. The reproduction does not exactly copy every feature of the original plot. For example, the multi-line spacing of the left axis labels do not conform, nor do the width and spacing of the bars or the type fonts or the black boxes of the left axis labels. That said, the revised plot has improved the chart components discussed above: the redundant representation of the numeric totals.

### Lawyer Resistance to Seeking Feedback from Their Clients

by James S. Wilber, Esq., Altman Weil, Inc.

As companies continue to pay careful attention to controlling costs, law departments remain under scrutiny. Accordingly, in-house lawyers regularly look for ways to demonstrate value to their clients. One of the easiest and most cost effective ways of doing this is to seek regular feedback from clients.

Some companies require support functions, such as law, to conduct annual satisfaction surveys, often in connection with performance evaluations of law department lawyers and staff. In many companies, however, law departments rarely, if ever, seek formal feedback from their clients about whether they are meeting client needs.

Lawyer personality data reveal that lawyers generally are averse to seeking feedback. This is not due to a lack of concern for clients, but rather to unique characteristics of the lawyer personality. Personality profile testing shows that most lawyers have particularly low resilience – the quality that determines how well one responds to criticism and rejection. Therefore, it is not surprising that asking for feedback is particularly difficult for lawyers.

If your department’s lawyers can overcome this barrier, they will find that merely asking clients for their opinions typically creates an enhanced image of the law department and an improved relationship between the department and its clients. This phenomenon seems to endure for several months to a year following the survey. Conducting satisfaction surveys seems to create a halo effect for the surveyor.

A 2002 study conducted by Paul M. Dholakia of Rice University’s Jesse H. Jones Graduate School of Management, and Vicki G. Morwitz of New York University’s Stern School of Business, supports the halo effect theory and concludes that merely conducting satisfaction surveys enhances client loyalty and profitability. [1]

Continually assessing and improving the level of service your law department provides will allow it to demonstrate its value to the company and meet the increasingly high demands of clients.

[1] The study, reported in the Journal of Consumer Research in September 2002, was highlighted in the May 2002 Harvard Business Review.

### Show foreign and domestic participants by level (Feb 23, 2015)

Having completed a client satisfaction survey, you can sort your participants all kinds of ways. Let’s give an example by sorting the participants by level and then also identifying them by their office location. Assume that all we do is code them by US office or international office.

The plot above shows the total number of participants by the light grey bar for each level. Within the bar there are two colored squares. The ones with a “T” on top indicate the number of participants who are US-based when you look straight down to the horizontal axis; those with an “F” indicate the number of foreign-based participants at that level.

### The Importance of Confidentiality in Law Department Client Surveys

by James S. Wilber, Esq., Altman Weil, Inc.

Most law departments periodically use written or online surveys to gauge the satisfaction of clients with the services they (and outside counsel) provide. A question that often arises is whether (and if so, how) to protect the confidentiality of the responses that come from the executives and other key users of legal services served by the law department?

The responses of clients to survey questions, and most especially the narrative comments that they provide, are of immense value to the general counsel and the law department in determining how to improve services and ensure the complete satisfaction of their clients. Although the tabulated results of the survey and the narrative comments are very helpful for these purposes even when the client responses are confidential and the law department cannot tell from whence they came, many general counsel prefer being able to follow up directly with individual clients to discuss their ideas and concerns. That of course isn’t possible if the names of the respondents are confidential. The competing factor is getting clients to be candid, something that is much more likely if the survey is confidential.

The best way to handle the issue of confidentiality versus candor is to promise confidentiality at the beginning of the survey but then ask the respondent to waive confidentiality at the end. Our experience is that participants usually are less concerned with keeping their responses confidential once they’ve actually provided them.

Therefore, at the beginning of the survey, make it clear that the identity and the responses of participants are confidential and will be unknown to the law department and all who work there.

To make the confidentiality promise real, the survey needs to be administered and tabulated by a third party – either an outside consultant or another group within the company that will conduct the survey for the law department.

At the very end of the survey, i.e., after the respondent has answered the questions and knows what was asked and how he or she responded, ask the client for a waiver of the confidentiality promised at the beginning. Clients are much more willing to waive confidentiality after they have completed the survey than before they know what questions will be asked and what their responses will be.

Make it clear that there is nothing wrong with the client refusing to waive confidentiality if that is the choice.

The final question should ask respondents whether they wish to have the responses remain confidential or whether they are willing to waive confidentiality and allow their individual questionnaires to be turned over to the general counsel.

Language we usually use to frame the final question is as follows: “Unless you waive confidentiality, Altman Weil requests your name for the sole purpose of tracking participation. The Law Department will see only tabulated results. Narrative remarks will be commingled without attribution. Your individual responses will remain confidential.”

Finally, at the very beginning of the survey it is important to define what is meant by the promise of confidentiality. Here is what we tell the clients of the law departments we help with client surveys: “To obtain candid responses to the survey, the source of any answer or comments will not be disclosed to the law department. Altman Weil will tabulate the questionnaires and report the tabulated responses as part of this study, but remarks will not be attributed to any individual. The only way in which the identity of a respondent might be discovered is if he or she words a comment in such a way that the source is obvious. Again, the source of your individual responses will be kept in strictest confidence.”

### Create a graphic that shows how many participants there were by each level

When you survey your clients to assess how satisfied they are with your law department, you want confidence that you have tapped a representative group by their levels. One way to visualize the responses you get from your survey by a distribution of levels is shown below. For this example, six ranks of clients are stacked from the lowest at the bottom, “Directors,” to the highest-level at the top, the “EVPs or above”.

Twenty-eight of the respondents were at the highest level of the company, followed by 74 Mgt Principals.

Yes, we could present the same information in a table (with the titles abbreviated):

Dir. Exec. Dir.   Manager   Sr. Mgr Mgt Princ.   EVP+

9      9           1         3       74         28

but a table lacks the visual dimension of length to convey relative numbers. A graph can provide that cue.

Our preference is a colorful graphic along the lines of the one shown below. The “arms” extending out from each level are the same length here, but you could, for example, show domestic and international on either side of the divide in the middle.

### Law Department Client Satisfaction

by James S. Wilber, Altman Weil, Inc.

Law departments that appreciate the value of client satisfaction understand the importance of obtaining systematic feedback from their clients. The methodological choice typically is between a written or online survey, on the one hand, or in-person meetings and interviews of clients on the other. The former method is easier, quicker, less expensive (in terms of opportunity cost) and provides a broader base of feedback. The latter method allows for a dynamic, iterative discussion where one answer or statement can lead to other important areas of inquiry, and of course, it can help improve relationships since it is done face-to-face. Our experience points to a hybrid approach being best – a survey of a large cross-section of key users of legal services, followed by in-person interviews of executives and top managers.

One advantage of doing an online survey is the ease with which metrics that will help you analyze the results can be generated. There are of course many, many metrics, slices of the data, etc., that can be produced. Set out below is one such example (specifically, a plot done from a metric), which shows a hypothetical company and its divisions and operating units.

### General Counsel of Fortune 500 companies, company revenue and law school rank

Let’s look at the Fortune 500 companies broken into equal-sized revenue categories and see if there is any pattern about the law school of their top lawyers.

The plot below shows 20 revenue categories increasing from \$4.8 billion for the smallest company on the left to \$89 billion and up in the farthest-right category. There are 25 companies in each category, plus or minus one.

The blue dots above a revenue category each represent one general counsel and the US News & World Report ranking of that person’s law school, which can be read from the vertical Y-axis. As noted before, when two schools have the same rank, this plot combines their graduate numbers. Unranked schools are not included and the law schools of a few general counsel are unknown.

It is apparent that the law schools with ranks from 1 to 50, which some might consider the elite law schools, produced many more general counsel than did the law schools in the two categories above that (51-100 and 101-150). On the other hand, the general counsel are quite evenly distributed across the various revenue ranges.

### A Shepard’s diagram of the three legal services companies can enlist

A Shepard’s diagram typically describes the composition of soil in terms of three materials: clay, silt, and sand. The pyramid labels each portion according to the respective proportions of those materials in a particular clump of soil. Naturally, seeing such a diagram led me to think about law departments.

My notion was that a company has three basic sources of legal guidance: its internal law department (if there is one), the law firms or private lawyers it hires, and the clients, consultants, outsource services and others who help identify, prevent, or deal with legal issues. In the diagram I call that third group, collectively, “resources”.

So, here is the Shepard’s diagram of the three legal services. Each side ranges from a low percentage of reliance on that service to a high percentage. So, for example, the top component represents a company where the law department does everything, there are no outside counsel, and the “resources” are mid-way. I tried to label the components that result in a short way, but realize that the labels can be improved.

Mostly, this diagram allows you to represent numbers tied into the relative contributions of law department service providers.

### Leveling off of law school rank and number of general counsel of Fortune 500 companies

Setting aside the number of students in each class of the ranked law schools, is there any pattern in the number of graduates who are Fortune 500 general counsel according to the law school’s rank? The plot above shows the number of graduates by the height of the blue dots and the left axis; the ranking of the schools ranges from number 1 on the left to 150 on the right. Bear in mind that when two schools have the same rank, this plot combines their graduate numbers. Unranked schools are not included.

The evident pattern appears to be that higher ranked schools have more F500 GCs, and the number of GCs per school declines as rankings go lower – until a third of the way across the numbers plateau. Your eye can fill in a line that starts high left, and drops down to about 30, at which point it levels off to the right side of the plot. From about a ranking of 30 on, the numbers bounce around randomly and fairly consistently. One might conclude from this specialized sample that if you didn’t graduate from a very elite law school, it doesn’t matter much if your goal is to reach the top rung of a Fortune 500 legal department.

A better way to depict the line would be with a trend line and confidence levels. But, that treat we hold for another post.

### Law schools and their graduates who are Fortune 500 GCs, adjusted for enrollment of law school

My hypothesis was that the larger the law school, in terms of students enrolled, the more graduates it would have who are the general counsel of a Fortune 500 company. To keep the data tractable and the plot below legible, I took the 150 top-ranked schools from US News & World Report and kept only the 34 schools that have at least three such graduates (some of the GCs do not have a law school associated with them).

Next, I divided that graduate number for each law school by the school’s enrollment (it is unclear as I write this whether enrollment includes LLM candidates, JSD candidates and other students aside from the standard three-year LLB or JD students). So that the numbers on the graph are not miniscule, I first divided the enrollment by 100.

Accordingly, the chart below presents the law schools on the bottom axis in alphabetical order and their Fortune 500 GC graduates per 100 enrollees above with a blue dot. Chicago and Harvard stand highest at two or more graduates per 100. Note that the former is a very large school; the other modest. At the other end, Brooklyn and Miami are lowest at about 0.3 (meaning for every three hundred enrolled students they have produced one Fortune 500 GC graduate).

My hypothesis fails, judging from the fairly random distribution of the dots.