Accessing the box
An extra step doesn't sound like much, but it has considerable consequences. This trade-off for easily sharing ownership across various functions or threads removes the ability to put as much data as possible into the CPU's cache, since the pointer makes any data locality difficult. Heap allocations themselves are expensive operations and reducing those will already provide a major speedup.
Furthermore, the compiler cannot deallocate a boxed value if it's still referenced in some places—a problem that occurs especially if the program is large and complex. Similar to orphaned objects in C# or Java, a saved Rc reference can easily be forgotten, creating a memory leak. Therefore, it's recommended to use heap memory only when required.
One principle piece of advice that requires a boxed value in Rust is to "favor object composition over class inheritance" (Gang of Four 1995:20). In the absence of class inheritance, the choice is obviously to use object composition. Considering that you should also "program to an interface not to an implementation" (ibid), there is often a strong wish to put a reference to a trait inside of a struct instead of directly working with the implementation.
To apply this architecture in Rust, the language requires us to put a trait's implementation into a Box<dyn TheTrait>, making it more difficult to handle, test, and reason about. This trait object requires the compiler to rely on dynamic dispatch, which is considerably slower than the default static dispatch.
Other than generics, there is no default solution to this issue. The impl Trait addition of Rust 2018 alleviates this issue for function parameters and return values, but cannot be used for field types.
So far, it looks like the best choice is to use concrete types instead of traits to avoid multiple dereference operations—as long as refactoring on change seems to be doable. If you create a library, generics are a better way to go for performance and flexibility.